#=SOURCE=# https://github.com/pytorch/pytorch/actions/runs/30529790819/job/90834162653
2026-07-30T10:55:18.2467887Z cachedir: .pytest_cache
2026-07-30T10:55:18.2468141Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2026-07-30T10:55:18.2468412Z rootdir: /var/lib/jenkins/workspace
2026-07-30T10:55:18.2468555Z configfile: pytest.ini
2026-07-30T10:55:18.2468817Z plugins: subtests-0.13.1, xdoctest-1.3.0, cpp-2.3.0, hypothesis-6.56.4, rerunfailures-14.0, flakefinder-1.1.0, xdist-3.3.1, typeguard-4.3.0
2026-07-30T10:55:18.2469101Z collecting ... collected 1 item
2026-07-30T10:55:18.2469255Z stepcurrent: Cannot find last run test, not skipping
2026-07-30T10:55:18.2469534Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather
2026-07-30T10:55:18.2469937Z 
2026-07-30T10:55:18.2470097Z distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather FAILED [310.0221s] [100%]
2026-07-30T10:55:18.2470292Z 
2026-07-30T10:55:18.2470367Z =================================== FAILURES ===================================
2026-07-30T10:55:18.2470578Z ________________ TestDistributedNNFunctionsGloo.test_all_gather ________________
2026-07-30T10:55:18.2470774Z Traceback (most recent call last):
2026-07-30T10:55:18.2479263Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 910, in wrapper
2026-07-30T10:55:18.2479572Z     self._join_processes(fn)
2026-07-30T10:55:18.2479872Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1180, in _join_processes
2026-07-30T10:55:18.2480173Z     self._check_return_codes(fn, elapsed_time)
2026-07-30T10:55:18.2480637Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1225, in _check_return_codes
2026-07-30T10:55:18.2480927Z     raise RuntimeError(
2026-07-30T10:55:18.2481107Z RuntimeError: Process 0 terminated or timed out after 310.0115718841553 seconds
2026-07-30T10:55:18.2481341Z ----------------------------- Captured stdout call -----------------------------
2026-07-30T10:55:18.2481548Z Timing out after 300 seconds and killing subprocesses.
2026-07-30T10:55:18.2481744Z ----------------------------- Captured stderr call -----------------------------
2026-07-30T10:55:18.2482050Z I0730 10:07:12.905000 80638 site-packages/torch/testing/_internal/common_distributed.py:989] Started process 0 with pid 80709
2026-07-30T10:55:18.2482421Z I0730 10:07:12.905000 80638 site-packages/torch/testing/_internal/common_distributed.py:989] Started process 1 with pid 80710
2026-07-30T10:55:18.2482828Z E0730 10:12:17.911000 80638 site-packages/torch/testing/_internal/common_distributed.py:1133] Could not retrieve traceback for timed out process: 0
2026-07-30T10:55:18.2485750Z E0730 10:12:22.917000 80638 site-packages/torch/testing/_internal/common_distributed.py:1133] Could not retrieve traceback for timed out process: 1
2026-07-30T10:55:18.2486257Z - generated xml file: /var/lib/jenkins/workspace/test/test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-91c55f19cb6af5f7.xml -
2026-07-30T10:55:18.2486609Z =========================== short test summary info ============================
#=MATCH=# 2026-07-30T10:55:18.2486986Z FAILED [310.0221s] ‹distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather› - RuntimeError: Process 0 terminated or timed out after 310.0115718841553 seconds
2026-07-30T10:55:18.2487349Z !!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
2026-07-30T10:55:18.2487533Z ======================== 1 failed in 310.03s (0:05:10) =========================
2026-07-30T10:55:18.2487756Z Exception ignored in atexit callback: <function _exit_function at 0x7fea0d9b9480>
2026-07-30T10:55:18.2487952Z Traceback (most recent call last):
2026-07-30T10:55:18.2488185Z   File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/util.py", line 357, in _exit_function
2026-07-30T10:55:18.2489944Z     p.join()
2026-07-30T10:55:18.2490140Z   File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/process.py", line 149, in join
2026-07-30T10:55:18.2490347Z     res = self._popen.wait(timeout)
2026-07-30T10:55:18.2490553Z   File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/popen_fork.py", line 43, in wait
2026-07-30T10:55:18.2490781Z     return self.poll(os.WNOHANG if timeout == 0.0 else 0)
2026-07-30T10:55:18.2491007Z   File "/opt/conda/envs/py_3.10/lib/python3.10/multiprocessing/popen_fork.py", line 27, in poll
2026-07-30T10:55:18.2491213Z     pid, sts = os.waitpid(self.pid, flag)
2026-07-30T10:55:18.2491346Z KeyboardInterrupt: 
2026-07-30T10:55:18.2491556Z Got exit code 1, retrying (retries left=1)
2026-07-30T10:55:18.2491695Z Command took >15min, returning 124
2026-07-30T10:55:18.2493864Z Test exited with non-zero exitcode 124. Command to reproduce: /opt/conda/envs/py_3.10/bin/python distributed/test_c10d_spawn_gloo.py --import-disabled-tests --import-slow-tests --use-pytest --save-xml test-reports/python-unittest -m (multigpu) --shard-id=1 --num-shards=1 -v -vv -rfEX -p no:xdist -x --reruns=0 --sc=distributed/test_c10d_spawn_gloo_1_404c2041c824f17d --print-items --pytest-single-test distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather
2026-07-30T10:55:18.2494798Z Test results will be stored in test-reports/python-pytest/distributed.test_c10d_spawn_gloo/distributed.test_c10d_spawn_gloo-93e3bc2e27058d1a.xml
2026-07-30T10:55:18.2495104Z ============================= test session starts ==============================
2026-07-30T10:55:18.2495341Z platform linux -- Python 3.10.20, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python
2026-07-30T10:55:18.2495553Z cachedir: .pytest_cache
2026-07-30T10:55:18.2495885Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow]
2026-07-30T10:55:18.2496142Z rootdir: /var/lib/jenkins/workspace
2026-07-30T10:55:18.2496268Z configfile: pytest.ini
2026-07-30T10:55:18.2496516Z plugins: subtests-0.13.1, xdoctest-1.3.0, cpp-2.3.0, hypothesis-6.56.4, rerunfailures-14.0, flakefinder-1.1.0, xdist-3.3.1, typeguard-4.3.0
2026-07-30T10:55:18.2498076Z collecting ... collected 1 item
2026-07-30T10:55:18.2498229Z stepcurrent: previously run test not found, not skipping.
2026-07-30T10:55:18.2498499Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all
2026-07-30T10:55:18.2498701Z 
2026-07-30T10:55:18.2498848Z distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all PASSED [3.7160s] [100%]
2026-07-30T10:55:18.2499037Z 
