Repository navigation
Description
Context
- Workflow:
multi_arch_release_linux_pytorch_wheels.yml(ROCm/rockrel) - Failing run: 42970099850, attempt 1
- Platform: Linux
- Affected architectures: gfx94X-dcgpu, gfx120X-all, gfx110X-all, gfx950-dcgpu, gfx90a, gfx1151
- PyTorch channel: nightly
- Python version: 3.11
- ROCm package version: 10.3.0a20261111
Summary
All six failed Test PyTorch jobs stop before test collection because pytest rejects --exclude-jit-executor. The error occurs in test_autograd.py, test_binary_ufuncs.py, test_cuda.py, test_nn.py, test_torch.py, and test_unary_ufuncs.py across these architectures. The missing JUnit XML errors follow the argument parsing failures; no individual test assertion is reached.
TheRock's external-builds/pytorch/run_pytorch_tests_full.py enables and forwards this option by default. The observed nightly test runner passes it to individual pytest commands, which reject it. The logs also warn that skip_tests/pytorch_2.16.py is absent, but the argument error is the immediate test blocker.
Exact Error
From the gfx110X job:
2026-11-11T10:04:53.3585286Z UsageError: usage: test_autograd.py [options] [file_or_dir] [file_or_dir] [...]
2026-11-11T10:04:53.3585930Z test_autograd.py: error: unrecognized arguments: --exclude-jit-executor
2026-11-11T10:04:53.3588380Z test_autograd.py: error: unrecognized arguments: --exclude-jit-executor
2026-11-11T10:04:53.3589536Z Test results will be stored in test-reports/python-pytest/test_autograd/test_autograd-c9b01f0a86343064.xml
2026-11-11T10:04:53.3598978Z FileNotFoundError: [Errno 2] No such file or directory: 'test-reports/python-pytest/test_autograd/test_autograd-c9b01f0a86343064.xml'
Failing jobs
| Architecture | Job |
|---|---|
| gfx94X-dcgpu | Test PyTorch |
| gfx120X-all | Test PyTorch |
| gfx110X-all | Test PyTorch |
| gfx950-dcgpu | Test PyTorch |
| gfx90a | Test PyTorch |
| gfx1151 | Test PyTorch |
Root cause commit
The jobs pin the wheel/test source to ROCm/pytorch@df21fc2d. ROCm/pytorch@5a989117 removed --exclude-jit-executor from test/run_test.py's argument parser and deleted the associated JIT executor test exclusions. The preceding nightly, 75b45b7c0c7409a0374c4a89a4a4549bbddf6af1, still accepts the option, while df21fc2d does not. The change originates in upstream pytorch/pytorch@6dca9f63 (#203621). TheRock's existing wrapper still sends the removed flag by default.
Proposed fix
Update TheRock's run_pytorch_tests_full.py to omit --exclude-jit-executor for PyTorch versions containing the removal and update its wrapper option/help text. If supported older release branches still need the exclusion, gate it by PyTorch version. A short-term validation is to run one failing module with the wrapper's --no-exclude-jit-executor option, then rerun the affected GPU jobs. This fix is proposed; it has not been tested here.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsTODO
Activity
amd-chiranjeevi commentedon Nov 11, 2026
Hi @jeffdaily @jithunnair-amd
Could someone check this issue ?
jithunnair-amd commentedon Nov 11, 2026
@ethanwee1 Looks like a straightforward fix, please put in a conditional fix based on PyTorch version if needed. Please do look into the git history to understand why this flag was removed from upstream and what the equivalent is, if any, and include that explanation in your fix PR