[qgallouedec]

Our GPU jobs run on aws-g4dn-2xlarge (T4, Turing), which has no bf16 and no FlashAttention-2 — CI exercises fp16/eager paths that no user runs — and so little VRAM that we are capped at 4 test workers.

	aws-g4dn-2xlarge (before)	aws-g6e-4xlarge (after)
GPU	1× Tesla T4	1× L40S
VRAM	15 GB	45 GB
Compute capability	7.5 (Turing)	8.9 (Ada)
bf16 / FA2	✗	✓
vCPU (physical cores)	8 (4)	64 (32)
RAM	30 GB	497 GB

Multi-GPU jobs are unchanged on aws-g5-12xlarge-cache (2× A10G, 23 GB each). One fix there: slow-tests.yml's multi-GPU job requested CUDA_VISIBLE_DEVICES: "0,1" while pinned to the single-GPU T4 group, so the slow multi-GPU suite was never actually running on 2 GPUs — it now points at the same multi-GPU group as tests.yml.

Observed speedup

~5× on the same job, same test set:

Job	Before (T4, 4 workers)	After (L40S, 32 workers)
Tests (3.13)	~46 min	9m22s
Tests without optional dependencies	~28 min	6m42s

Why hardcode 32 workers?

The Makefile now pins pytest -n 32 instead of -n auto.

Pinning keeps the worker count independent of whichever runner picks up the job. 32 is what -n auto resolved to on this one.

Why 32 is the right ceiling: CI historically ran 4 workers on 15 GB, i.e. ~3.8 GB of VRAM per test. 32 workers on 45 GB is ~1.4 GB per test — tighter, not looser — and it passes today. So transient OOM is a real possibility. If we hit it, the preferred fix is to chase the few high-VRAM tests; failing that (less preferable) we drop 32 → 24 → 16.

Known failures: the kernels-community/flash-attn2 tests, marked xfail

test_train_padding_free (SFT + DPO) and test_vlm_training (GRPO, slow) are gated on is_ampere_or_newer(), so they have always been skipped on the T4 and are only now running for the first time. They fail for reasons unrelated to the hardware, both in the kernel:

• The kernel version transformers currently resolves publishes build variants for torch 2.7–2.12 only, and our env installs torch 2.13, so the model never loads: flash-attn2: no build variant for torch 2.13 kernels-community#1082.
• The newer v3 stable-ABI build does load on torch 2.13, but its backward pass raises for any GQA model, on any torch version — including the tiny-Qwen2ForCausalLM-2.5 these tests train: flash-attn2 v3 (stable ABI): backward fails for GQA/MQA kernels-community#1085.

Marked non-strict xfail, so they will simply XPASS once the kernel is usable again.

---

Note

Low Risk
Changes are limited to CI runner labels and expected-failure markers on a few tests; no library training or runtime behavior is modified.

Overview
Moves single-GPU CI from aws-g4dn-2xlarge (T4) to aws-g6e-4xlarge (L40S) across tests.yml, slow-tests.yml, tests-experimental.yml, and tests_transformers_branch.yml, so jobs run on Ada hardware with bf16 and Flash Attention–capable GPUs instead of Turing fp16/eager paths.

Fixes slow multi-GPU CI in slow-tests.yml by pointing run_all_tests_multi_gpu at aws-g5-12xlarge-cache (matching other multi-GPU workflows) instead of the single-GPU T4 group while still setting CUDA_VISIBLE_DEVICES: "0,1".

Leaves tests_latest.yml on T4 with a TODO to switch to L40S after v1.11 ships, because the pinned v1.10-release branch lacks the new flash-attn2 xfail markers and would fail on Ada.

Marks three Ampere-gated kernel tests as xfail (test_train_padding_free in SFT/DPO and test_vlm_training in GRPO) now that they actually run on L40S; failures are attributed to upstream kernels-community/flash-attn2 issues (torch 2.13 builds and GQA backward), not the hardware move.

^{Reviewed by Cursor Bugbot for commit fc41207. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 970072b139

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Excessive hardcoded pytest workers

Medium Severity

make test now hardcodes pytest -n 32 instead of -n auto. The target CI host is aws-g6e-4xlarge with 16 vCPUs (and typically ~8 physical cores for -n auto), so this oversubscribes CPU and piles more workers onto a single 48 GB GPU. The same target is used locally, so developer machines also get forced to 32 workers. That raises OOM/flake risk—especially with OutOfMemoryError already in the rerun filter and existing GPU cleanup aimed at xdist pressure.

 

^{Reviewed by Cursor Bugbot for commit 934b761. Configure here.}

[qgallouedec]

see desc

[albertvillanova]

The rationale in the description is about runner stability, but make test is the command we give contributors (CONTRIBUTING.md:166), so this also forces 32 workers on laptops with far fewer cores!

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8e394f7. Configure here.}

[albertvillanova]

Nice speedup, and thanks for fixing the multi-GPU group.

I forced the re-run of the failing CI workflows.

Two things before I approve: see below.

[albertvillanova]

The rationale in the description is about runner stability, but make test is the command we give contributors (CONTRIBUTING.md:166), so this also forces 32 workers on laptops with far fewer cores!

[albertvillanova]

Thanks.

For the record about:

• Restore -n auto in make test; it resolves to 32 on the CI runner anyway

32 it's guaranteed, not incidental: xdist prefers psutil.cpu_count(logical=False) for auto, and psutil always comes in through accelerate, so it resolves to physical cores in every job (including the .[test]-only one), 32 on this runner.