[qgallouedec]

Removes trl.experimental.grpo_with_replay_buffer (GRPOWithReplayBufferTrainer, GRPOWithReplayBufferConfig, ReplayBuffer), added in #4060.

Why

High maintenance cost. GRPOWithReplayBufferTrainer subclasses GRPOTrainer and overrides _generate_and_score_completions, the surface that churns most (grpo_trainer.py: 208 commits since the replay buffer landed). The override is a hand-copied version of that method, and it has drifted: 385 of 616 lines differ from the GRPO original today.

31 commits touched trl/experimental/grpo_with_replay_buffer/: GRPO work re-applied by hand inside the copy: #4157 and #4203 (images), #4391 (image insertion), #4300 (agent training), #6001 and #6002 (multiple environments), #6238 (environment-owned reward), #6083 (MoE aux loss), #6352 (log_multimodal), #6439 (truncated-completion loss normalization), #6693 (None logprobs from vLLM). #4158 refactored GRPO and RLOO specifically to make the override possible.

Low adoption. 8 genuine trainings in the last 2 months as per our telemetry, ~1 year after landing (2025-09-24).

Notes

trl.experimental has no stability guarantee, so no deprecation cycle. The code stays in git history for anyone wanting to revive it.

---

Note

Medium Risk
Breaking change for the small set of imports of GRPOWithReplayBufferTrainer; core GRPO training is unchanged.

Overview
Removes the experimental trl.experimental.grpo_with_replay_buffer package, including GRPOWithReplayBufferTrainer, GRPOWithReplayBufferConfig, and ReplayBuffer, along with its dedicated tests and documentation page.

Public surface is trimmed by dropping GRPOWithReplayBufferTrainer from the experimental trainer list in docs (index.md, _toctree.yml) and from the telemetry allowlist in base_trainer.py. Users who relied on this trainer must switch to standard GRPOTrainer or recover the code from git history; trl.experimental had no stability guarantee, so there is no deprecation shim.

^{Reviewed by Cursor Bugbot for commit 2b6f306. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-09T14:24:07.497872Z	e2c4bf9	PR opened

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cmpatino]

LGTM!