Skip to content

[Issue] Linux PyTorch nightly tests fail across six GPU families: pytest rejects --exclude-jit-executorΒ #9899

Open

Description

@amd-chiranjeevi

Context

  • Workflow: multi_arch_release_linux_pytorch_wheels.yml (ROCm/rockrel)
  • Failing run: 42970099850, attempt 1
  • Platform: Linux
  • Affected architectures: gfx94X-dcgpu, gfx120X-all, gfx110X-all, gfx950-dcgpu, gfx90a, gfx1151
  • PyTorch channel: nightly
  • Python version: 3.11
  • ROCm package version: 10.3.0a20261111

Summary

All six failed Test PyTorch jobs stop before test collection because pytest rejects --exclude-jit-executor. The error occurs in test_autograd.py, test_binary_ufuncs.py, test_cuda.py, test_nn.py, test_torch.py, and test_unary_ufuncs.py across these architectures. The missing JUnit XML errors follow the argument parsing failures; no individual test assertion is reached.

TheRock's external-builds/pytorch/run_pytorch_tests_full.py enables and forwards this option by default. The observed nightly test runner passes it to individual pytest commands, which reject it. The logs also warn that skip_tests/pytorch_2.16.py is absent, but the argument error is the immediate test blocker.

Exact Error

From the gfx110X job:

2026-11-11T10:04:53.3585286Z UsageError: usage: test_autograd.py [options] [file_or_dir] [file_or_dir] [...]
2026-11-11T10:04:53.3585930Z test_autograd.py: error: unrecognized arguments: --exclude-jit-executor
2026-11-11T10:04:53.3588380Z test_autograd.py: error: unrecognized arguments: --exclude-jit-executor
2026-11-11T10:04:53.3589536Z Test results will be stored in test-reports/python-pytest/test_autograd/test_autograd-c9b01f0a86343064.xml
2026-11-11T10:04:53.3598978Z FileNotFoundError: [Errno 2] No such file or directory: 'test-reports/python-pytest/test_autograd/test_autograd-c9b01f0a86343064.xml'

Failing jobs

Architecture Job
gfx94X-dcgpu Test PyTorch
gfx120X-all Test PyTorch
gfx110X-all Test PyTorch
gfx950-dcgpu Test PyTorch
gfx90a Test PyTorch
gfx1151 Test PyTorch

Root cause commit

The jobs pin the wheel/test source to ROCm/pytorch@df21fc2d. ROCm/pytorch@5a989117 removed --exclude-jit-executor from test/run_test.py's argument parser and deleted the associated JIT executor test exclusions. The preceding nightly, 75b45b7c0c7409a0374c4a89a4a4549bbddf6af1, still accepts the option, while df21fc2d does not. The change originates in upstream pytorch/pytorch@6dca9f63 (#203621). TheRock's existing wrapper still sends the removed flag by default.

Proposed fix

Update TheRock's run_pytorch_tests_full.py to omit --exclude-jit-executor for PyTorch versions containing the removal and update its wrapper option/help text. If supported older release branches still need the exclusion, gate it by PyTorch version. A short-term validation is to run one failing module with the wrapper's --no-exclude-jit-executor option, then rerun the affected GPU jobs. This fix is proposed; it has not been tested here.

Activity

  1. amd-chiranjeevi commented on Nov 11, 2026

    @amd-chiranjeevi
    ContributorAuthor

    Hi @jeffdaily @jithunnair-amd
    Could someone check this issue ?

  2. jithunnair-amd commented on Nov 11, 2026

    @jithunnair-amd
    Contributor

    @ethanwee1 Looks like a straightforward fix, please put in a conditional fix based on PyTorch version if needed. Please do look into the git history to understand why this flag was removed from upstream and what the equivalent is, if any, and include that explanation in your fix PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions