[qgallouedec]

Closes #5903.

What

When an environment_factory is provided, the environment now owns the data. train_dataset becomes optional: reset() returns the prompt.

This removes the need to fabricate a dummy dataset just to drive the loop (procedural envs like chess, self-sampling envs like Wordle).

Applies to both GRPOTrainer and AsyncGRPOTrainer.

Behavior

• No train_dataset + environment_factory → a placeholder dataset of empty prompts is synthesized internally
• No train_dataset + no environment_factory → raises (unchanged intent).
• A provided dataset no longer needs a "prompt" column: any missing prompt defaults to an empty message that reset() supplies. This makes multi-environment routing datasets that carry only an environment column work (the env owns the prompt).
• Providing a train_dataset still works exactly as before: its rows feed reset(**kwargs) and its length drives the loop.

Group-state note

Because the environment self-samples on each reset(), the G members of a group may not share the same initial state (slightly noisier baseline). Passing a group_id to reset() for deterministic shared state is left to a follow-up.

---

Note

Medium Risk
Changes core GRPO/AsyncGRPO data and prompt wiring; behavior is guarded and tested, but env-owned self-sampling can make group baselines noisier when group members get different reset() states.

Overview
train_dataset is optional for GRPOTrainer and AsyncGRPOTrainer when a single environment_factory is set: the environment supplies tasks via reset() (return string = user prompt), and max_steps defines how long training runs. The trainer synthesizes a small placeholder dataset of empty prompts to drive the dataloader.

Validation: no dataset and no environment still errors; dict multi-env factories still require a routing dataset with an environment column; env-owned training without positive max_steps errors.

Rollout path: with environments, missing prompt on a row defaults to an empty user message before reset() observation is applied—so routing datasets can be environment-only. Without an environment, missing prompt still fails.

Reward models: conversational detection for nn.Module reward funcs now uses is_conversational({"prompt": prompts[0]}) in GRPO, RLOO, and SDPO (fixes prompts-only batches).

Docs reframe env-first training, optional external datasets, and multi-env routing examples; tests cover guards and env-owned training without a zen prompt dataset.

^{Reviewed by Cursor Bugbot for commit 9cc8d5c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c6edf17a2a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[adithya-s-k]

Approving the direction from the Harbor / OpenReward integration side — this is the right move, and the backward-compatible path (an external dataset still drives length; prompt just becomes optional) drops into both specs without breaking them. Filed #6354 to adopt environment-owned reward + dataset there on the back of this.

Two non-blocking follow-ups from that angle:

1. Self-sampling + num_generations > 1 quietly degrades the GRPO baseline. You flag this in the group-state note, but it's a real footgun: with an external dataset the G rollouts of a group share one task, so advantages are computed over the same initial state; with env-owned self-sampling each reset() can draw a different task, so the group's baseline is taken across different problems. Harbor/OpenReward sidestep it by staying dataset-driven (one task per row → shared across the group), which this PR preserves. For the new path, a warn when there's no dataset and num_generations > 1 (until group_id lands) would catch it — it won't show up in the loss curve.

2. No-dataset + dict environment_factory (#6002) don't compose. The placeholder is {"prompt": ...} with no environment column, so with a routing dict the first batch hits raise ValueError("Example has environment=None ...") — confusing, since the user passed no examples. Since env-owned data implies a single owning env, worth rejecting a dict factory upfront in the no-dataset branch (or stating single-factory-only in the docs).

Neither blocks — great ergonomics win for procedural / self-sampling envs.

[qgallouedec]

Thanks! On the two follow-ups:

• Agreed it's a footgun, kept it in the docs note for now; the runtime warning would false-alarm on deterministic envs (e.g. chess), so I'll fold it in with group_id.
• Fixed: a dict factory without a dataset now raises upfront (routing needs the environment column) instead of the confusing environment=None error. We can consider supporting this case though: randomly sample an environment? let's see depending on the real use cases

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ea7ed9e. Configure here.}

[AmineDiro]

Is this possible ? because we set placeholder dataset :

   train_dataset = Dataset.from_dict({"prompt": [[{"role": "user", "content": ""}]] * num_placeholder_rows})

[qgallouedec]

Yes; the placeholder isn't what this branch is for. It covers "no dataset at all", and those rows always have a prompt. This handles the other case: a user-provided dataset + environment_factory with no prompt column, e.g. a routing dataset carrying only environment. The environment_factories is not None half keeps a genuinely malformed dataset failing fast. Same shape as GRPOTrainer after 6013034.

[AmineDiro]

I wonder how this will play out with 🤔
#6385

[qgallouedec]

right, they compose: #6385 says an explicit max_steps > 0 disables the epoch callback, and the env-owned path mandates max_steps > 0. That's this guard. So the callback never engages here.
Good thing too: #6385's target is ceil(num_train_epochs × |dataset|), and our |dataset| is just placeholder rows sized to one generation round, so that target would be meaningless. #6385 should add a test pinning "no dataset ⇒ epoch callback inactive".

[AmineDiro]

All good, just wanted to point out the interaction with #6385 for AsyncGRPOTrainer