Repository navigation
Can support optional dynamic sampling in GRPO ? (as in DAPO) #4764
Description
Feature request
Hi TRL team,
I would like to ask whether it would be acceptable to add optional dynamic resampling / dynamic sampling support to GRPOTrainer, similar to the mechanism used in DAPO-style GRPO training.
Motivation
The motivation is to handle cases where all sampled completions for a prompt receive identical rewards, leading to degenerate groups with zero advantages. Dynamic sampling would trigger additional rollouts only in such cases, while keeping a strict upper bound and leaving the default behavior unchanged.
Your contribution
If the team considers this feature suitable for inclusion, I would be happy to try implementing it and submit a PR.
I am still relatively early in my experience with TRL’s codebase, so I cannot guarantee the implementation will be perfect, but I am willing to iterate based on feedback and align with the project’s standards.
Activity
pramodith commentedon Dec 30, 2025
@ceresOPA you may want to checkout the experimental GRPOWithReplayBufferTrainer.
It replaces groups with 0 advantage with already seen examples with a high variance.
Add DAPO Dynamic Sampling support for GRPO and RLOO trainers
qgallouedec commentedon May 24, 2026
Hey @ceresOPA, sorry for the slow reply; catching up on the backlog. This one's been requested a few times already: #3708 (Jul 2025) and #3973 (Sep 2025) both ask for DAPO-style dynamic sampling in GRPO, with a draft attempt in #3758. Closing as a duplicate so the discussion stays on the canonical thread. Please follow / +1 / contribute over at #3708. In the meantime, as @pramodith mentioned, the experimental
GRPOWithReplayBufferTrainer(trl.experimental.grpo_with_replay_buffer) addresses the same degenerate-zero-advantage problem via a replay buffer rather than resampling, and it's available today if you want to try it.