What's Changed

• Keep a parenthesis inside an LFM2.5 tool-call argument from closing the call by @albertvillanova in #7942

Full Changelog: v1.17.0...v1.17.1

Features

🔥 Mistral Large 4: fine-tune a 1T-parameter model on one node

TRL now supports mistralai/Mistral-Large-4-1T-A52B: 1T parameters, 52B active. The new LoRA example fine-tunes it with SFTTrainer on a single 8×B300 node, on top of its FP8 weights: the experts are split 8 ways with token dispatch, everything else is sharded across the 8 GPUs at load time by transformers' DistributedConfig, and LoRA trains attention and the dense and shared-expert MLPs.

Measured: 32.7k tokens per step in ~8 s, 142 GiB per GPU, with packing and assistant-only loss on reasoning conversations.

training_args = SFTConfig(
    model_init_kwargs={
        "dtype": "auto",  # keep the FP8 weights
        "distributed_config": DistributedConfig(fsdp_size=8, ep_size=8),
    },
    packing=True,
    assistant_only_loss=True,
    ...
)
trainer = SFTTrainer(model="mistralai/Mistral-Large-4-1T-A52B", args=training_args, peft_config=LoraConfig(...), ...)

assistant_only_loss=True works out of the box: TRL ships a training chat template that matches the released checkpoint's, including its reasoning_effort levels (low, medium, high). Needs transformers>=5.22.0, peft>=0.21.0 and accelerate>=1.15.0.

• Add Mistral Large 4 by @qgallouedec in #8017
• Sync the Mistral Large 4 chat template with the released checkpoint by @albertvillanova in #8018

🎯 Unbiased importance sampling under top-k, top-p and min-p

vLLM normalizes its log-probabilities over the tokens left after top_k, top_p and min_p filtering, while the trainer normalized over the whole vocabulary. Every sampled token's log-ratio was off by the log of the probability mass the filter kept, so the vLLM importance sampling ratio was biased towards 0 (around 1e-4 instead of 1 with top_p=0.9). The trainers now normalize over the candidate set each token was sampled from, following Keep Sampling Mask from DeepSeek-V3.2.

• AsyncGRPO: renormalize log-probs over the top-k/top-p candidate set by @AmineDiro in #7627
• GRPO: renormalize log-probs over the top-k/top-p candidate set (vLLM colocate mode, vLLM >= 0.28; server mode will follow) by @albertvillanova in #7866
• Require top_k with min_p in AsyncGRPO so the importance sampling ratio stays unbiased by @albertvillanova in #7937
• Track the support maximum in the chunked log-prob kernel, so a far-below-max candidate set no longer gives a log-probability of 0 and non-finite gradients, by @kashif in #7983

💥 What this changes for you. vLLM bounds the candidate set it returns with top_k, so with vLLM and the importance sampling correction on (the default), GRPOConfig now raises when top_p < 1.0 or min_p > 0.0 is set without top_k > 0. Set top_k (e.g. top_k=512), or vllm_importance_sampling_correction=False.

⚖️ Score centering is the default AsyncGRPO objective

AsyncGRPO's old log-probs are the vLLM sampler's, so its ratio mixes staleness with the numerical mismatch between the server and the trainer. With an FP8 sampler and a bf16 trainer, the clipped objective collapses: reward goes from 0.60 to 0.34 (FP8 weights) or 0.10 (FP8 weights + KV cache) by step 300, against 0.71 with a bf16 sampler.

Score centering with masked importance sampling is now the default (score_centering=True, importance_sampling_min=0.5, importance_sampling_max=5.0). With it, the FP8 sampler trains exactly like the bf16 one (0.72). It costs 15–20% of generation throughput.

💥 What this changes for you. AsyncGRPO runs now train with a different objective by default, and tokens whose ratio to the sampler falls outside [0.5, 5.0] are dropped. Set score_centering=False to keep the clipped objective.

by @AmineDiro in #7760

🚀 Faster AsyncGRPO rollouts and packing

• Adaptive in-flight cap from the vLLM load: the rollout worker follows the server's KV cache usage instead of admitting a fixed max_inflight_tasks, so requests stop queueing and being preempted on the server. No new config: everything is read from the server. By @AmineDiro in #7610
• Filter zero-advantage groups (DAPO's dynamic sampling): groups whose rollouts all got the same reward, or none of which could be scored, no longer cost a forward and backward. By @AmineDiro in #7609
• Pack the sampling mask ragged and pad it on the device: TreeRow.pack goes from 250 s to 31 s on rank 0 over a 15-step run, and from 1040 ms to 66 ms on a 28k-token multi-turn row. By @AmineDiro in #7798
• Keep filling a micro-batch when another row has free room by @clementtisseau in #7793
• Use FlashAttention-4 on Blackwell GPUs in async GRPO and async distillation by @lewtun in #7792

🌳 Tree attention for linear-attention hybrids

Tree packing (v1.16) now supports the Gated DeltaNet hybrids: Qwen3.5, Qwen3.6, Qwen3-Next and OLMo-Hybrid. Since v1.16 the trainer refused them at init; before it, sequence packing accepted them but was silently wrong on them. The causal convolution and the delta-rule recurrence now follow the tree instead of the row, through FLA's chunked kernels.

by @AmineDiro in #7686

🎚️ Precise chunked log-probabilities

The chunked log-probability and KL/JSD heads get the two precision fixes from A Fix for Softmax and Cross-Entropy: confident targets keep their small gradients, and matching teachers produce zero gradients. Entropy-preserving reinforcement learning (Appendix B.4) reports the same high-confidence policy-gradient issue.

• Fix chunked loss precision by @kashif in #7882
• Fix the fused log-prob/entropy backward on strided inputs (shifted views from SFT/DPO) by @winglian in #7890

🔀 X-Token cross-tokenizer distillation in GOLD

GOLDTrainer gets off-policy cross-tokenizer distillation based on X-Token: set lmbda=0.0 and xtoken_loss_type="p_kl" or "h_kl" with a precomputed projection matrix.

• Add X-Token cross-tokenizer KD to GOLD trainer by @kashif in #6126
• Support byte-fallback tokenizers (Gemma) in GOLD byte-offset alignment by @kashif in #8010

🟥 Training on AMD GPUs

TRL's trainers now run on AMD Instinct GPUs through PyTorch's ROCm build, with a guide and an AMD CI job that runs the test suite on every PR.

• Document training on AMD GPUs by @qgallouedec in #7844
• Add AMD/ROCm CI by @kashif in #5918

🧩 More models

• Beam: tiny model, SFT/DPO/GRPO/RLOO tests, GRPO tool calling, and a LoRA SFT example on one 8×B300 node. Needs transformers>=5.20.1. By @qgallouedec in #7629
• Mistral Medium 3.5: training chat template for assistant-only loss by @qgallouedec in #7940
• LFM2.5-24B-A2B (and LFM2.5-2.6B's current template) for tool calling by @qgallouedec in #7753
• LFM2.5-1.2B-Instruct and LFM2.5-VL-450M for tool calling by @albertvillanova in #7909, #7910 and #7911
• TRL's own Qwen3 training template for response parsing, so a tokenizer saved by assistant-only SFT works in GRPO, by @edbeeching in #8007

Other

• Accept instantiated models in AsyncGRPOTrainer by @kashif in #6676, and port instantiated models, MoE aux loss and the injected-worker LoRA name to AsyncDistillationTrainer by @kashif in #7984
• Save AsyncGRPO and AsyncDistillation checkpoints in the mixed-precision dtype instead of fp32 by @AmineDiro in #7939
• Back off between unscorable harness sessions and stop the worker when they persist by @albertvillanova in #7992
• Async Harbor GRPO: PEFT support and fixes by @younesbelkada in #7849 and #7948
• Write the logged completions to parquet as JSON by @qgallouedec in #7696
• Add support for vLLM 0.32.0 by @qgallouedec in #7799, and allow its patch releases by @albertvillanova in #7850

Removals and requirement bumps

• Remove the experimental TPO trainer by @qgallouedec in #7807
• **Require top_k with top_p < 1.0 or `min...

Features

🛠️ TRL now trains on AWS Trainium

The fused LM head from v1.15 is what made Neuron tractable; the remaining gaps are closed here. There is a guide to go with it.

Neuron recompiles once per input shape, so an unpadded run stalls about a minute on every new sequence length. pad_to_multiple_of now defaults to "auto": 128 on Neuron, None elsewhere, so nothing changes on GPU. Trainium2, Qwen3-0.6B SFT, 20 steps: 130 s with the default against 260 s without.

• Add a guide for training on AWS Trainium by @qgallouedec in #7729
• Default pad_to_multiple_of to "auto" by @qgallouedec in #7724
• Run the fused LM head and distillation kernels on devices without Triton by @qgallouedec in #7669
• Report the Neuron device in trl env and telemetry by @qgallouedec in #7672
• Support any accelerator in activation offloading by @qgallouedec in #7670

🌳 Tree attention for AsyncGRPO

AsyncGRPO now packs every micro-batch row as a prefix tree instead of a flat sequence. Multi-turn and multi-generation rollouts repeat a lot of prefix: on a shared prompt, those tokens used to be forwarded once per row. Tree packing forwards them once.

rollout 0:  [ prompt ] [ completion A ]              prompt ──┬── completion A ── later turn
rollout 1:  [ prompt ] [ completion B ]        =>             └── completion B
rollout 2:  [ prompt ] [ completion A ] [ … ]

Chains without a fork collapse into segments, and each segment's queries attend to the K/V gathered along its root-to-segment path in a single FA3 varlen call. No mask tensor: every layer except attention runs once per unique token, and only K/V is duplicated along shared prefixes.

Qwen3-1.7B, 32K tokens per rollout, forward + backward:

There is no packing option to choose: with nothing shared a tree row is exactly the old sequence packing, at a ~1% cost.

by @AmineDiro in #7328

Muse Spark Open

Support for meta-models/Muse-Spark-Open-816B-A42B: the tiny-model script, coverage in the VLM tests (SFT, DPO, KTO, chat-template utils, fused LM head), and a LoRA SFT example on 8 nodes of 8×H100 with DistributedConfig(fsdp_size=64, ep_size=8), token dispatch and packing. Measured at 14.2 s/step, 131k tokens/step, 40.2 GB per GPU.

Needs transformers>=5.20.0; the adapter save additionally needs transformers#49500 with peft 0.21.2.

by @qgallouedec in #7536

AMD GPUs in the async trainers

• Support AMD GPUs in the async GRPO and async distillation trainers by @Abdennacer-Badaoui in #7689
• Hide the GPU from the async rollout worker on ROCm by @Abdennacer-Badaoui in #7685

Tool-call arguments typed by the tool's JSON schema

parse_response now receives the tools, so tool-call arguments come back typed from the tool's own schema instead of whatever the raw parse produced.

by @qgallouedec in #7650

Other

• Async GRPO tool metrics by @AmineDiro in #7624
• Store async GRPO and distillation rollout sample tokens as numpy arrays by @AmineDiro in #7626
• Save the final step when async trainers stop at num_train_epochs by @lewtun in #7628
• Pass sequence boundaries from the padding-free collator by @qgallouedec in #7599
• Build the reference model with the policy's dtype and attention implementation by @qgallouedec in #7597, and the SDFT/SDPO teacher with the student's by @qgallouedec in #7616
• Find unused parameters under DDP for vision-language models by @qgallouedec in #7598
• Pad the generation prompts to pad_to_multiple_of by @qgallouedec in #7671
• Drop the client-side observe loop from the CARLA examples by @sergiopaniego in #7721
• Use None for unscored env.reward in OpenReward by @albertvillanova in #7541

Removals and requirement bumps

• Remove the experimental SSD trainer by @qgallouedec in #7706
• Refuse nn.DataParallel in RewardTrainer by @albertvillanova in #7571
• Drop vLLM 0.21.0 support by @qgallouedec in #7700
• Require accelerate>=1.10.1 and drop the ParallelismConfig shim by @albertvillanova in #7643
• Declare a minimum bitsandbytes version by @albertvillanova in #7641
• Declare the direct dependencies TRL gets transitively by @albertvillanova in #7713

Fixes

• Fix silently wrong torch.compile results in the Triton loss kernels on torch < 2.11. torch_compile=True trained with loss=0.0 and grad_norm=nan at every step. Inductor passes scalar kernel args as constexpr before torch 2.11, the .to(tl.float32) casts raised on those, and Inductor then assumed every input was mutated and miscompiled. By @albertvillanova in #7688
• Fix FSDP2 vLLM colocate weight sync for tied-embedding models by @Abdennacer-Badaoui in #7647
• Prevent VLM policies from generating the image placeholder token in GRPO, RLOO and Distillation by @albertvillanova in #7695
• Fix Online DPO VLM training when the prompt exceeds max_length (#7703) and for processors with 5D pixel values (#7704), both by @albertvillanova
• Fix activation offloading on torch < 2.4 by @albertvillanova in #7748
• Fix torch.compile(fullgraph=True) of selective_log_softmax_and_entropy on torch 2.6 by @albertvillanova in #7719
• Keep the Triton loss kernels in fp32 under torch.compile by @qgallouedec in #7600
• Read is_triton_available once in the fused LM head kernels so Dynamo does not warn by @qgallouedec in #7731
• Fix Distillation for models split across devices by @albertvillanova in #7573
• Restore the gradient checkpointing kwargs after generation by @qgallouedec in #7668
• Drop the FlashAttention warnings for padding-free and packing in SFT by @qgallouedec in #7620
• Restore the distributed env vars that vLLM colocate tests leave behind by @Abdennacer-Badaoui in #7723

Clearer errors

A sweep making failure modes explicit instead of crashing deep in the stack, all by @albertvillanova:

• Raise a clear error when the fused LM head is used without Triton (#7639)
• Raise a clear error for prefix tuning on peft versions that cannot support it (#7646)
• Raise a clear error when the Distillation student and teacher heads are on different devices (#7605)
• Reject a text-only teacher in GOLD VLM distillation instead of crashing (#7581)
• Name the missing vLLM server setting when the server cannot serve TRL (#7611)
• Raise early for use_kernels with transformers 5.2.0 and 5.3.0 (#7577), and if kernels is installed with transformers 5.3.0 in the tiny NemotronH scripts (#7575)

Documentation

• Add a guide for training on AWS Trainium by @qgallouedec in #7729
• Add guidance on testing through the public API to the contributing guide and agent instructions by @albertvillanova in #7606
• Document the criterion for raising a dependency version floor by @albertvillanova in #7638

CI

Real objects, not mocks. The bulk of this cycle, applying the repo's testing rule across the suite, all by @albertvillanova: run the GRPO entropy mask tests on a real trainer instead of a Mock (#7566), the teacher logprob tests on a real trainer instead of a MagicMock (#7579), build real GOLD trainers in the init (#7582), input preparation (#7614), on-policy generation (#7613, #7615), output decoding (#7633), ULD helper (#7634) and no-backend tokenizer (#7635) tests, check GOLD dataset preparation through the trainer (#7632), and build real async trainers in the rollout state checkpoint tests (#7583)

Coverage, all by @albertvillanova: test SFT (#7552) and DPO/KTO/Reward/GRPO/RLOO (#7601) on a model ...

Features

⚡ Fused LM head: up to 6.9× longer sequences on the same GPU

SFT, DPO, KTO, GRPO, RLOO and Distillation now score tokens with a fused LM head: a Triton kernel projects the hidden states through the LM head in tiles and reduces to per-token log-probs and entropy directly, so the [batch, seq, vocab] logits tensor is never built. It is on by default, there is nothing to turn on.

Max trainable sequence length (gemma-3-1b, 262k vocabulary, 79 GiB of GPU memory, bf16, batch size 1, gradient checkpointing, sdpa):

At 8,192 tokens, peak memory drops 52% to 82% and steps are 2.3% to 10.9% faster:

The gain is largest for small models with large vocabularies. SFT's previous default (chunked_nll) already avoided full logits, so it is unchanged. Scoring needs Triton on a GPU (Linux with CUDA, ROCm or XPU).

💥 What this changes for you. The full-logits scoring path, the use_liger_kernel chunked path and _forward_redirection are gone from GRPO, RLOO, DPO and KTO:

• A PEFT adapter on lm_head now raises. Use modules_to_save=["lm_head"] instead.
• use_liger_kernel=True is deprecated in these four trainers and will be removed in v2.0.0. Liger's layer kernels still apply, but fused_linear_cross_entropy is forced to False (it would replace the fused head) and setting it explicitly raises. Use model_init_kwargs={"use_kernels": True}.
• In DPO and KTO, compute_metrics and compute_loss(..., return_outputs=True) still receive the full logits, from an extra forward pass taken only when they are used. Both warn that this is deprecated and will be removed in v2.0.0.

Nothing else the trainers logged is lost: the kernel gained log_sum_sq_probs (WPO weighting), mean_logits (logits/chosen, logits/rejected) and is_top1 (DPO's mean_token_accuracy) as opt-in outputs, each checked against the full logits in tests.

• Compute the chunked log-probabilities with a Triton kernel by @qgallouedec in #7386
• Add an opt-in fused LM head to the patched forward by @qgallouedec in #7387
• 💥 Score GRPO and RLOO tokens with the fused LM head by @qgallouedec in #7389
• 💥 Score DPO and KTO tokens with the fused LM head by @qgallouedec in #7390
• Score SFT tokens with the fused LM head by @qgallouedec in #7466
• Compute the distillation divergence with a Triton kernel by @qgallouedec in #7465
• Compute only the requested fused LM head outputs by @qgallouedec in #7406
• Rename patch_fused_lm_head to add_fused_lm_head by @qgallouedec in #7415
• Honor logits_scaling and lm_head_multiplier in the fused LM head by @YaseenBashaT in #7439
• Fix the fused LM head for models split across devices by @albertvillanova in #7479

Selective activation checkpointing in SFT

With gradient checkpointing on, the attention output is now saved during the forward instead of being recomputed in the backward, recovering most of the checkpointing slowdown at long context for one extra hidden-state-sized tensor per layer. Same eager SAC approach torchtitan uses under FSDP2, no torch.compile needed.

SFTConfig(
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"selective": True},
)

PEFT + DeepSpeed ZeRO-3 is rejected: that combination needs reentrant checkpointing, which SAC cannot use.

by @kashif in #7194

Assistant-only loss on vision datasets in SFT

assistant_only_loss=True now works on vision datasets. SFTTrainer used to refuse it because processors did not return usable assistant masks once image placeholders were expanded; transformers#48793 fixed that, and it ships in transformers 5.18.

SFTConfig(assistant_only_loss=True)  # now fine with an image dataset

The vision collator tokenizes conversational examples through the processor's apply_chat_template and requests the assistant masks to mask non-assistant tokens in the labels. Older transformers still raises, now with the required version in the message.

by @albertvillanova in #7377

Conversations in the completions table

Prompts and completions are logged as raw conversation lists instead of batch_decode'd flat strings, so the table renders turn by turn. Much more readable for multi-turn and tool-calling setups, and for VLMs.

The rendered system prompt (including tool definitions injected by the chat template) is no longer visible, since the original messages are shown rather than the fully-templated string.

by @qgallouedec in #5309

AsyncGRPO and AsyncDistillation

• AsyncGRPO: train OpenEnv harnesses from validated token captures by @adithya-s-k in #6947
• Add PEFT support to AsyncDistillationTrainer by @kashif in #7476
• Fix duplicate completions in AsyncGRPO groups with data-parallel vLLM by @lewtun in #7549

Other

• Add SmolVLM original and training chat template with generation markers by @aazizyan in #5868
• Declare gradient accumulation loss scaling with loss_is_scaled_for_ga by @qgallouedec in #7508
• Keep SFT's MoE aux loss independent of world size and gradient accumulation by @qgallouedec in #7570
• Discover environment tools on the class, not the instance by @Rome-1 in #7360
• Accept an IPv6 base_url in VLLMClient by @qgallouedec in #7462
• Pass processor kwargs via processor_kwargs in the VLM SFT collator by @qgallouedec in #7547
• Call trainer.end() at the end of the scripts and examples by @qgallouedec in #7384
• Install vLLM in the TRL Docker images by @qgallouedec in #7464
• Train harnesses through Harbor: drop the standalone opencode example by @sergiopaniego in #7457
• Add support for vLLM 0.31.0 by @qgallouedec in #7567

Breaking changes and removals

• Drop Python 3.10 support by @qgallouedec in #7497
• Refuse nn.DataParallel (and drop the multi-GPU slow test job) by @qgallouedec in #7407
• Remove the experimental MiniLLM trainer by @qgallouedec in #7500
• use_liger_kernel is deprecated in DPOTrainer, KTOTrainer, GRPOTrainer and RLOOTrainer and will be removed in v2.0.0. These trainers now use the fused LM head, so only Liger's layer kernels apply; use model_init_kwargs={"use_kernels": True} instead. Docs updated by @qgallouedec in #7535
• Drop vLLM 0.20.0 / 0.20.1 / 0.20.2 support by @qgallouedec in #7435, #7499 and #7568
• Remove the unreachable prepare_multimodal_messages_vllm by @behroozazarkhalili in #6946

Fixes

• Fix chat template drift: recognize the current LFM2 (#7494) and LFM2.5 (#7493) chat templates, the other Gemma 4 revisions (#7495), parse the current LFM2 tool calls (#7522), and read templates shipped as named variants by their default variant (#7530), all by @albertvillanova
• **Fix SFTTrainer not appending EOS when `data...

Patch release fixing two cases of silently wrong training and three crashes.

Worth calling out: #7505 affects GRPO, RLOO and Distillation on the default generation path (use_vllm=False). Models that end a turn with an id only their generation config declares (Gemma 3/4, Phi-3.5, ...) did not stop at the end of the turn, and everything generated after it was trained on, up to max_completion_length. The only visible sign was completions/clipped_ratio close to 1. With vLLM, generation stopped correctly but the metrics were still wrong, and mask_truncated_completions=True dropped every finished completion. Runs on models whose tokenizer eos is their only end-of-turn id are unchanged.

What's Changed

• Keep the DFT loss finite on a batch with no trainable tokens by @albertvillanova in #7471
• Fix precompute_ref_log_probs under FSDP with no reference model by @qgallouedec in #7507
• Fix: Start the completion where the tokenized prompt and prompt+completion diverge by @qgallouedec in #7463
• End completions on every eos id the model declares in GRPO, RLOO and Distillation by @albertvillanova in #7505
• Guard the requests and urllib3 imports in the vLLM client by @qgallouedec in #7545

Full Changelog: v1.14.1...v1.14.2

What's Changed

• Fix the server-mode crash at the first weight sync on vLLM 0.20 to 0.25 by @albertvillanova in #7410
• [Fix] Generate one completion per sample after tool calls in vLLM server mode by @qgallouedec in #7418
• Preserve explicit model_init_kwargs in CLI scripts by @DaoyuanLi2816 in #7394
• Honor resume_from_checkpoint in CLI scripts by @DaoyuanLi2816 in #7391

Full Changelog: v1.14.0...v1.14.1

Features

trl.losses is gone: DPO, KTO and GRPO stream their own log-probs

v1.13 vendored Liger's chunked_loss into trl.losses as a holding action, not a destination (#7063). The problem it was holding: those classes reimplement each trainer's loss math, so TRL carried two copies of every formula, and copies drift. That drift is where the bugs were. GRPO's grad_norm differed from the default path, KTO ignored the reference model under PEFT, KTO class weights and DPO label_smoothing were silently dropped. Worst of all, under plain DDP the fused path ran the loss on the unwrapped model, so DistributedDataParallel's reducer was never armed and gradients were never all-reduced: every rank silently kept its own.

The fix is to stop reimplementing. All DPO, KTO and GRPO need from the fused path is per-token log-probs of the selected tokens, and TRL already had _ChunkedLogProbFunction for exactly that: it streams the vocabulary with an online logsumexp and recomputes in the backward pass. Each trainer now runs backbone -> _ChunkedLogProbFunction -> its own existing loss code, unchanged. One implementation of each loss, and the memory win is kept, because full logits are still never materialized.

Thirteen restrictions existed only because the loss had been rewritten in a form that could not express those options. Most are gone: DPO with use_liger_kernel=True now accepts mixed loss types, f-divergences and precomputed reference log-probs, and GRPO gains entropy and off-policy masking. Still refused: use_weighting, compute_metrics, a PEFT adapter on lm_head, prompt-learning PEFT, and the MoE auxiliary loss.

Single H100, Qwen3-0.6B, bs=4, seq 512:

The third row is the point: that configuration did not exist before, because the fused path rejected precomputed reference log-probs outright.

use_liger_kernel=True still works and still enables Liger's model kernels through transformers. In DPO, GRPO and KTO it now selects TRL's chunked log-probability path instead of Liger's fused loss.

• Stream KTO log probabilities by @kashif in #7075
• Stream GRPO log probabilities by @kashif in #7077
• Move DPO to chunked log probabilities by @kashif in #7243
• Drop the fused Liger JSD path from the experimental distillation trainers by @kashif in #7301
• Skip frozen chunked log probability gradients by @kashif in #7076
• Redirect the KTO liger loss through the distributed wrapper under DDP by @qgallouedec in #7247
• Document what use_liger_kernel actually does in each trainer by @qgallouedec in #7361

A fused Triton logprob + entropy kernel now ships in TRL

The loss head is the only hot path TRL owns: transformers already kernelizes the model internals, and nothing on the Hub covers what happens after the decoder. On one H100 (bf16, V=151936, H=4096, 8192 tokens), selective_log_softmax plus entropy_from_logits cost 12.8 ms and 2.32 GiB over [8, 1024, 151936]. A fused Triton kernel does the same in 0.89 ms, roughly 14x, and lands closer to the fp32 reference than the bf16 path it replaces. GRPO runs this two to four times per step.

It now lives in-tree at trl.kernels and is on by default on CUDA, ROCm and XPU, with the PyTorch implementation as fallback. It was briefly loaded from the Hub; that was reversed because this kernel is pure Triton and compiles at runtime, so the Hub bought nothing while costing a version gate unrelated to whether the kernel works, a download that fails offline, trust_remote_code=True for anyone with kernels installed, a silent except Exception: pass wrapped around all of it, and 13 of 13 tests skipped in CI.

• Use fused kernels for logprobs and entropy by @kashif in #7253
• Move TRL loss kernels into TRL by @kashif in #7266
• Ship the logprob and entropy kernel in trl instead of the Hub by @qgallouedec in #7363

Other

• Ensure device agnostic for examples/docs/templates by @kaixuanliu in #6770
• AsyncGRPO: PEFT/LoRA support with adapter-only vLLM sync by @AmineDiro in #7017
• Add support for vLLM 0.29.0 by @qgallouedec in #7178
• [GOLD] Recommend LFM2.5 and Gemma 4 students by @kashif in #6388
• Add TailSFT example by @kashif in #7212
• [AsyncGRPO] Run sync tools on a thread pool and support async tools by @AmineDiro in #7175

Breaking

Six experimental trainers removed

GRPOWithReplayBufferTrainer, BCOTrainer, PRMTrainer, XPOTrainer, NashMDTrainer and the GSPO-token trainer are gone from trl.experimental.

What changed is not the guarantee, which was always that trl.experimental has none, but the fact that the decision is now written down. Removals used to be argued one pull request at a time, with every thread relitigating the policy before it got to the feature. #7182 settled what the judgment weighs, and #7254 put it in the docs: usage, external issues and pull requests that come from people actually running the feature, whether a stable trainer already covers it, maintenance cost, downstream consumers, owner, and age. Deliberately inputs to a judgment rather than a test: no threshold, no notice cycle, and the numbers stay in the removal pull request where you can check them. A paper index entry outlives the code it described.

The numbers are what make the case. BCOTrainer was 1,600 lines carrying its own reference-model handling, DeepSpeed preparation, tokenization and eval loop, for 11 genuine trainings in two months; it was also the only reason TRL depended on scikit-learn and joblib at all, and both are now gone from pyproject.toml. GRPOWithReplayBufferTrainer had hand-copied _generate_and_score_completions from GRPO and drifted to 385 of 616 lines different, for 8 trainings.

• Remove the experimental BCO trainer by @qgallouedec in #7136
• Remove the experimental GRPO-with-replay-buffer trainer by @qgallouedec in #7132
• Remove the experimental PRM trainer by @qgallouedec in #7133
• Remove the experimental Nash-MD trainer by @albertvillanova in #7255
• Remove the experimental XPO trainer by @albertvillanova in #7256
• Remove the experimental GSPO-token trainer by @albertvillanova in #7331

Also breaking, and covered above: trl.losses was removed along the way.

• Move DPO to chunked log probabilities by @kashif in #7243

Fixes

• Gradients were never all-reduced for KTO under plain DDP. compute_loss redirected through the distributed wrapper only for ZeRO-3 and FSDP, so on plain DDP the loss ran on the unwrapped model, DistributedDataParallel.forward() never ran, its reducer was never armed, and every rank silently kept its own gradients. By @qgallouedec in #7247
• Wrapped packing duplicated data when the input table was sliced. _pack_wrapped sized the output from the sliced view but read from the raw child buffer, so it re-read from the start of the buffer instead of from the slice, and datasets.map(batched=True) passes sliced tables. By @qgallouedec in #6670
• A reward function logging on only some ranks hung or crashed the run. GRPO and RLOO looped over the keys each rank held and called one collective per key, so ranks called different collectives: log_metric on rank 0 alone hung on NCCL, and log_extra crashed inside gather_object with a 1EB allocation. By @behroozazarkhalili in #7382
• Special token ids now reach the config the Trainer actually reads, so it no longer realigns them at train time and logs a change the user did not make (part of #7093): eos in the trainers (#7127), written to the text config for composite models (#7315), the reward model pad token (#7318), and the requested pad token applied to vision datasets (#7316, #7352)
• Do not quantize the teacher by default in the distillation and GOLD entry points by @behroozazarkhalili in #6769
• Prepare the dataset once per node when the datasets cache is node-local by @qgallouedec in #7241
• Read the MoE auxiliary loss coefficient from the model config by @qgallouedec in #7248
• Await async reward functions written as callable classes by @albertvillanova in #7213
• Fix two stream races in OffloadActivations by @kashif in #7195
• Only dedupe offloaded activation storages in streams mode by @kashif in #7193
• Pass the FSDP2 ignored_params kwarg only...

Features

Training beyond 1M tokens

A new long-context guide plus a runnable example that trains a book-length sequence per step on a single 8×H100 node.

Measured: Qwen3-8B at 1,048,576 tokens, 380 s/step, 56.2 GB per GPU (bf16, per_device_train_batch_size=1, loss_type="chunked_nll"). Runs end to end and saves. The guide documents the levers in the order you hit them (chunked_nll, gradient-checkpointing offload, YaRN RoPE) and the constraints (full attention only, no packing). Needs transformers >= 5.16 for checkpointing offload.

• Document 1M-token training and add a context-parallelism example by @qgallouedec in #6846
• Rewrite the long context guide by @qgallouedec in #7003

Chunked-CE lm_head projection now runs on tensor cores

The inner loop of the default loss_type="chunked_nll" was doing h.float() @ w.float().t(). Both operands are already bf16, so the upcast bought no information: it moved the GEMM off the tensor cores onto the fp32 SIMT path and materialized an fp32 copy of the whole lm_head weight (2.03 GB for a 248k vocabulary), rebuilt for every chunk and again on every gradient-checkpoint recompute. In an 8×H100 profile of trl sft on Qwen3.6-35B-A3B those two fp32 SIMT GEMMs were 21.6% of all GPU kernel time.

One chunk, 256 tokens × vocab 248,320 × hidden 2048, 1×H100, bf16, fwd+bwd:

End to end, 16,384 tokens per step, tokens/s/GPU:

Same fix applied to the distillation trainers (which paid it twice per chunk, student and teacher). Under accelerate mixed precision the numerics are bit-identical (autocast was already casting the operands back down); without autocast over the projection the GEMM moves to bf16 tensor cores, which is exactly what loss_type="nll" already does.

by @qgallouedec in #6863

Fused linear losses vendored into trl.losses

TRL's only import from liger-kernel was liger_kernel.chunked_loss (the fused linear DPO / KTO / GRPO / JSD losses). That module is pure PyTorch, and most of the GRPO and DPO variants in it were written for TRL. Upstream reviews had stalled, so the code comes home as trl.losses: FusedLinearDPOLoss, FusedLinearKTOLoss, FusedLinearGRPOLoss, FusedLinearJSDLoss (copied from Liger-Kernel v0.8.2, BSD-2 notice kept, bitwise identical to the installed Liger on random inputs).

No new config flag: use_liger_kernel=True keeps selecting the fused loss exactly as before, and still needs liger-kernel installed because transformers' Trainer patches the model kernels with it.

• Vendor the fused linear losses from Liger-Kernel into trl.losses by @qgallouedec in #7059
• Fix DPO and KTO Liger loss gaps by @kashif in #7062
• Harden chunked log probabilities by @kashif in #7065
• Drop fused JSD from DistillationTrainer by @kashif in #7064
• Test liger vs non-liger GRPO loss parity by @kashif in #6814
• Run the vendored loss parity suites as slow tests only by @qgallouedec in #7067

Contributor skills at .agents/skills

Agent skills for contributing to TRL now ship in the repo at .agents/skills.

by @qgallouedec in #6901

Docs: stable vs experimental taxonomy, and how to change the objective

• Split the docs taxonomy into stable and experimental trainers by @qgallouedec in #7022
• Document how to change the training objective by subclassing a trainer by @sergiopaniego in #6918
• Document how to choose and pin a Hub kernel version by @albertvillanova in #6998
• Add missing TRL blog posts to the docs landing page by @qgallouedec in #7023

QLoRA on the quantization_config argument

The QLoRA test suites move onto the quantization_config trainer argument added in v1.8, and gain coverage on the VLM paths.

• Add a QLoRA test to the GRPO and RLOO VLM test suites by @albertvillanova in #6909
• Add a QLoRA test to the trainers that support quantization_config by @albertvillanova in #6910
• Convert the QLoRA tests to the quantization_config trainer argument by @albertvillanova in #6927

Dependency floors raised

• Require peft>=0.13.0 and drop the 0.12 version guards by @qgallouedec in #7102
• Require deepspeed>=0.18.6 and drop the 0.16.4 guard by @albertvillanova in #7118
• Guard target_parameters use for peft<0.17.0 by @albertvillanova in #7119
• Add support for vLLM 0.28.0 by @qgallouedec in #7001
• Drop vLLM 0.19.0 support by @qgallouedec in #7002

Other

• Count each token once in num_tokens under tensor parallelism by @qgallouedec in #7101
• XPO / NashMD perf: avoid output_hidden_states when only last_hidden_state is used by @ciaoyizhen in #4755
• Sync the model config pad token id with the tokenizer in the trainers by @albertvillanova in #7097 and (experimental) #7128
• Forward model revision to the auto-loaded processing_class by @kashif in #6978
• Inject the model revision into concrete config loads too by @albertvillanova in #7099
• Match hub kernels by repo id when checking the attention implementation by @albertvillanova in #6985
• Catch AsyncDistillationTrainer up with recent AsyncGRPO changes by @kashif in #6823 and #6987
• Remove the leftover dataset_formatting module by @albertvillanova in #7117
• Remove the unused FLASH_ATTENTION_VARIANTS constant from the DPO trainer by @albertvillanova in #7013
• Document that padding-free is temporarily unavailable in DPO by @albertvillanova in #7014
• Realign duplicated comments across the trainers by @qgallouedec in #7145

Breaking

⚠️ PPO is removed

PPOTrainer, PPOConfig, and modeling_value_head.py (PreTrainedModelWrapper, AutoModelForCausalLMWithValueHead, AutoModelForSeq2SeqLMWithValueHead) are gone, along with their tests, doc page, and examples.

Bit of a moment for TRL: PPOTrainer predates every PR in this repo. It landed in dfb6a580 on 2020-03-28, the commit that first added the library, back when the package was called lm_ppo. It was the oldest thing in TRL and the last piece of the original codebase.

Why now: unmaintained (no feature work in over a year, every PPO commit since #4482 a drive-by fix), the only trainer never aligned on the input format (it still took tokenized input_ids), near-zero recorded usage, and a magnet for automated bug hunters filing real reports against code nobody runs. Nobody loses anything: from trl import PPOTrainer already stopped working in v1.10, so anyone actually running PPO is pinned to an older TRL and those installs keep resolving exactly as they do today.

create_reference_model stays (BCO, A2PO and Online DPO use it).

by @qgallouedec in #7020

Fixes

• Fix missing entropy gradient in _ChunkedLogProbFunction backward — backward accepted grad_entropy but never used it, so any loss backpropagating through the entropy output silently got zero gradient from that term, with no error or warning. Only AsyncGRPOTrainer read entropy (under no_grad, for logging), so no user-facing regression today, but this is exported non-experimental code and an entropy-bonus loss term would have hit it silently. By @verma8076 in #6625
• vLLM weight sync now fails instead of hanging by @AmineDiro in #6913
• Fail fast on incompatible vLLM for default Async GRPO weight transfer by @DimensionSTP in #6830
• Fix VLM support in AsyncGRPO by @AmineDiro in #6839
• Fix variable reference in BCO docs by @albertvillanova in #7046
• Fix the IW-OPD self-distillation loss assertion in tests by @albertvillanova in #7123
• Fix typos in help strings and er...

⚠️ v1.12.0 is an accidental duplicate of v1.11.0

During the v1.11.0 release, VERSION was bumped to 1.12.0 instead of 1.12.0.dev0. Our publish workflow fires on any push to main that touches VERSION and only skips the upload when the version contains "dev", so trl 1.12.0 was auto-published to PyPI at 19:46 on 2026-08-26, 90 seconds after 1.11.0, with identical code.

trl==1.12.0 on PyPI is bit-identical to trl==1.11.0. There are no features, fixes, or behavior changes in this version.

The version number is burned (PyPI does not allow yanking + re-uploading the same version), so we skip 1.12 entirely for real development. The next feature release is v1.13.0.

Full Changelog: v1.11.0...v1.12.0 (empty)

Features

Use vLLM's own server instead of TRL's

trl vllm-serve used to run a custom FastAPI app around vLLM's LLM, with its own data-parallel fan-out and a worker extension for weight sync. vLLM's own server covers all of that. The custom server is gone: trl/scripts/vllm_serve.py drops from 1218 to ~130 lines and now translates its flags into vllm serve, prints the equivalent command, forwards any extra argument, and sets the three settings TRL needs (--weight-transfer-config, --logprobs-mode processed_logprobs, --max-logprobs -1). fastapi, uvicorn and pydantic leave the vllm extra.

Weight sync now uses vLLM's NCCL weight-transfer engine — tensors are announced once and streamed in a single packed broadcast, instead of one HTTP request + broadcast per tensor. Distillation teacher logprobs come from prompt_logprobs.

Benchmark (GRPO server-mode on H100, Qwen2.5-1.5B / Qwen2.5-VL-3B, same seed, same data):

Bit-identical completions vs the old server (16/16 over 4 prompts × n=4, logprobs identical to 0.0 over 499 tokens). trl vllm-serve still accepts the same flags — it just forwards to vllm serve now.

by @qgallouedec in #6765

New experimental AsyncDistillationTrainer (+ multi-teacher MOPD)

Async on-policy distillation, architected like AsyncGRPOTrainer: a background rollout worker generates the student's own completions and scores them against a teacher served over HTTP, so generation and training overlap instead of alternating. The teacher is never loaded locally — just a vLLM server URL.

Also supports MOPD (multi-teacher on-policy distillation): pass more than one entry in teacher_server_urls and route each sample to a teacher via a teacher_id column (e.g. a math teacher and a code teacher, each served independently).

from trl.experimental.async_distillation import AsyncDistillationConfig, AsyncDistillationTrainer

trainer = AsyncDistillationTrainer(
    model="Qwen/Qwen3-4B",
    args=AsyncDistillationConfig(
        teacher_server_urls=["http://teacher-math:8000", "http://teacher-code:8000"],
        teacher_top_k=16,
    ),
    train_dataset=dataset,  # rows carry a `teacher_id` column for MOPD
)
trainer.train()

by @kashif in #6705

New model & template support

• Muse Glimmer (+ chat template) by @qgallouedec in #6738 and #6739
• Nemotron 3.5 Lightning by @qgallouedec in #6736
• LFM2.5-VL by @qgallouedec in #6737
• Qwen3.8 by @qgallouedec in #6764
• DeepSeek-R1-Distill chat template by @qgallouedec in #6766

Examples reorganized into per-example folders

examples/ was split by format (scripts/ vs notebooks/), which scattered related files (e.g. sft_nemotron_3 existed as both a script and a notebook, in different folders; OpenEnv notebooks lived apart from OpenEnv scripts). Everything is now one folder per story — the name says the method and the task (grpo_wordle, sft_gpt_oss, ppo_tldr), and the folder holds every file the example needs (scripts, notebooks, prompts, chat templates, eval code). Thin single-trainer demo scripts are removed (they live in trl/scripts/ and each trainer doc page has a runnable snippet). Old examples/scripts/… and examples/notebooks/… GitHub / Colab links will 404 — worth flagging in downstream posts.

by @qgallouedec in #6820

AsyncGRPO: metrics rework, dtype control, and cleanup

• Rework AsyncGRPO metrics + docs by @AmineDiro in #6715
• Training dtype in AsyncGRPOConfig by @AmineDiro in #6774
• Simplify AsyncGRPOTrainer — drop redundant metric guards, merge train-begin callbacks by @qgallouedec in #6378
• Fix checkpoint resume triggering unnecessary vLLM inference by @AmmarHassona in #5911

Rewritten TRL skill (Python-API first) + drop trl skills install

The TRL skill is now written around the Python API — trainer selection, dataset format, load-bearing config fields, GRPO reward-function contract — with the CLI as a footnote. trl skills install is dropped and the skill source moves out of the package.

• Rewrite the TRL skill around the Python API by @qgallouedec in #6794
• Drop trl skills install and move the skill source out of the package by @qgallouedec in #6793

DistillationTrainer supports tool calling

Continuing the graduation cycle from v1.10.

by @qgallouedec in #6723

Suggest migrating to DistillationTrainer when the GKD config is fully covered

GKDTrainer now emits a hint when the config is fully covered by the stable DistillationTrainer — no behavior change, just a nudge for users still on the older API.

by @qgallouedec in #6847

Other

• Allow dict config fields to be set from the command line by @qgallouedec in #6792
• Raise when packing is combined with context parallelism by @qgallouedec in #6843
• Emit position_ids from the SFT collator when sequence parallelism is enabled by @qgallouedec in #6812
• Remove one host sync from SFT chunked_nll loss by @qgallouedec in #6842
• Align the chunked-loss config access between SFT and Distillation by @qgallouedec in #6744
• Add dataset_source to telemetry by @qgallouedec in #6599
• Raise a clear error when generation returns no completions by @albertvillanova in #6714
• Move GPU CI from T4 to L40S by @qgallouedec in #6741
• Add on-policy multimodal GKD regression test by @sergiopaniego in #6342

vLLM version sweep

• Add support for vLLM 0.27.0 and 0.27.1 by @qgallouedec in #6742
• Drop vLLM 0.17 by @qgallouedec in #6722
• Drop vLLM 0.18 by @qgallouedec in #6844
• Require liger-kernel 0.8.2 and drop the SAPO warning filter by @albertvillanova in #6768

Fixes

• Fix GRPO/RLOO training with multiple images for VLMs without grid metadata by @qgallouedec in #6570
• Fix GRPO/RLOO with dynamic-tiling VLMs (InternVL) by @qgallouedec in #6619
• Handle None token logprobs from vLLM in GRPO importance sampling by @behroozazarkhalili in #6693
• [GRPO] Match the Liger dapo / cispo / vespo normalizer to the non-Liger path — completes the fix from v1.10 (#6024) on the Liger path. By @kashif in #5890
• [SFT] Treat past_key_values and attentions as optional on the model output by @Hakureirm in #6686
• Fix AsyncGRPOTrainer crashing on environment-owned datasets by @albertvillanova in #6824
• Fix SDPO scale_rewards CLI parsing and warn on ignored feedback by @sergiopaniego in #6341
• Fix opencode.py proxy trace / log paths after OpenEnv moved them to functions by @sergiopaniego in #6672

Documentation and Examples

• Rework AsyncGRPO docs alongside the metrics rework (@AmineDiro, #6715)
• Reorganize examples into self-contained per-example folders (see above)
• Suggest migrating to DistillationTrainer when GKD config is covered (see above)

CI

Big CI-hygiene cycle by @albertvillanova. Highlights:

• Test one Python version on PRs, matrix nightly by @qgallouedec in #6763
• Notify Slack when the tests of the latest release fail by @albertvillanova in #6900
• Post the Docker image build results to the shared CI Slack channel by @albertvillanova in #6903
• Factor the Slack notification into a local composite action by @albertvillanova in #6895 (+ updates in #6896, #6897, #6908, #6934, #6936)
• Move slow-test job status handling, dependencies, and reporting to Slack by @albertvillanova in #6752, #6754, #6756
• Pin the TruffleHog scanner image by digest + tighten secret scanning by @albertvillanova in #6677, #6757, #6758, #6759
• Various hotfix pins and reverts across the cycle (bitsandbytes / liger-kernel / peft dev builds / transformers dev) by @albertvillanova in #6...