Repository navigation
REQUEST: Dynamic Sampling for GRPO #3973
Description
Dear developers,
It would be great if we can have dynamic sampling (from DAPO paper) in TRL.
Almost every other mainstream rl libraries (VeRL, ms-swift,...) now have implemented dynamic sampling. I understand that this request has been raised several times before, but I would still think it is valuable to have them. Maybe it is less about the efficiency of training, but more about training stability and final reachable performances. (I think it is fair to have somewhat slower training, but with better final performance in return, because it seems to me dynamic sampling is just a disguised curriculum learning under the hood)
I understand that given the current codebase, it requires quite complex code change to implement it, but I think it is still worthwhile to raise this request again.
Thank you for your great work and efforts!
Activity
almeidava93 commentedon Aug 29, 2025
I started working on dynamic sampling and implemented a very simple version that I ran in my setup. Here's the link to the PR: #3758
I stopped working on it, because I'm not able to test efficiency in bigger models. I believe that better/bigger models may benefit more from this strategy, since the probability of a correct answer is not too low.
I tested with a 1B parameter model, which has very limited capability in the math datasets used for training in the DAPO paper. Since the probability of a correct answer with this model I used is pretty small, the training loop needed a lot of resampling until a better answer was generated. I believe that is the main thing that made it inefficient.
Of course, my implementation was a simple resampling until a non zero std appeared and it needs adaptation for bigger batches as well.
I'm happy to help if HF team defines this as worth it.
wenquanlu commentedon Aug 30, 2025
Thanks for your awesome implementation!
Yea based on the current codebase, I think resampling is the most realistic to implement, but I believe using the buffer approach would be much more efficient. However, I do understand this requires very complex code change. (As I have no idea how to do it myself). So I just raise this request again to see if whether the HF team would approve.
abc8350712 commentedon Sep 22, 2025
Does trl support dynamic sampling now?
almeidava93 commentedon Sep 22, 2025
Add DAPO Dynamic Sampling support for GRPO and RLOO trainers
qgallouedec commentedon May 29, 2026
Hey @wenquanlu, sorry for the slow reply, catching up on the backlog. DAPO-style dynamic sampling for GRPO has been requested in a few threads now; #3708 (Jul 2025) is the oldest, with a draft attempt in #3758 by @almeidava93. Closing this one as a duplicate so the discussion stays on the canonical thread; please +1 / follow / contribute over at #3708. In the meantime, the experimental
GRPOWithReplayBufferTrainer(trl.experimental.grpo_with_replay_buffer) tackles the same zero-advantage-group problem via a replay buffer instead of resampling, and is available today if you want to try it.