[Thanatos9404]

What does this PR do?

Fixes #4613.

count:counting_composition cannot be passed by a response that follows its instruction. The instruction asks to "Write 3 paragraphs, delimited by the markdown divider: * * *, with exactly {n_sent} sentences each, with exactly {n_words} words in each sentence", but the checker in verifiable-instructions (f46a5ac, the head of its default branch, which the unpinned requirements.txt installs):

1. splits paragraphs with re.split(r"\s?\*\*\*\s?", value) (L2440), so a * * * divider is never found and the response counts as one paragraph;
2. counts words with nltk.word_tokenize(sentence) (L2457), and split_into_sentences keeps the period that ends each sentence, so Cats sleep. is three words.

What it does accept is *** dividers with sentences one word short, because the period fills the last word slot. This affects all 2,131 of the 46,391 rows in nvidia/Nemotron-RL-instruction_following that carry this constraint. The checker is a copy of open-instruct's IFEvalG one, which had the same bug and was fixed in allenai/open-instruct#1718.

Changes:

• counting_composition.py (new): CountingCompositionChecker subclasses the upstream class and overrides only check_following. The instruction text, default arguments and get_instruction_args() are inherited, so the published prompts still describe what is graded. The rules match the open-instruct fix:
  • paragraphs are split on a line that holds * * * (spaces or tabs around the stars and CRLF line endings are accepted; asterisks inside a paragraph are not a divider);
  • an empty segment before the first divider or after the last one is ignored, while an empty paragraph between two dividers fails (upstream meant to allow the outer ones but still ran them through the sentence count, so they failed);
  • a word is an nltk.word_tokenize token with at least one letter or digit, so punctuation is no longer counted.
• app.py: a module-level INSTRUCTION_DICT is the upstream registry with this one entry replaced, and verify() looks instruction classes up there. Every other instruction still uses the upstream checker.
• README.md: one paragraph saying this instruction uses a local checker and why.

Why a local override: verifiable-instructions has had no commits since 2025-08-20 and its only open PR (from January) has had no maintainer response, so there is no fixed commit to pin. Changing the instruction text to *** would not help the published dataset, whose 2,131 prompts all say * * *. Contiguous *** stays rejected, since the instruction names * * * and *** is the divider that paragraphs:paragraphs asks for. The issue suggests \s*\*\s*\*\s*\*\s*; that also accepts *** and an inline * * *, so this follows open-instruct's line-anchored rule instead (allenai/open-instruct#1966 is the same report against IFEvalG, closed by #1718). Keeping open-instruct's rules also keeps scores comparable between the two.

Unchanged on purpose: contractions still count as two words (Don't tokenizes to Do and n't), as in open-instruct.

Not in this PR: count:count_unique has the same punctuation problem. It compares the word_tokenize tokens with their set, so any response with two sentences that both end in . repeats the . token and fails. open-instruct fixed it in the same PR; it can follow separately.

Validation

Ubuntu (WSL), Python 3.13.14, CPU only. Server venv built from resources_servers/instruction_following/requirements.txt (verifiable-instructions resolved to f46a5ac), reused through .venv with +skip_venv_if_present=true:

gym env test +entrypoint=resources_servers/instruction_following +should_validate_data=true +skip_venv_if_present=true
# 30 passed
# The data for resources_servers/instruction_following has been successfully validated!

pre-commit run --all-files
# all hooks passed, no files changed (guardwords-check printed "private catalog unavailable" and skipped)

New tests in tests/test_app.py. The counting_composition cases go through verify():

• test_counting_composition_positive (5 cases): the issue's example; , ! ? punctuation; 3 sentences of 3 words with blank lines around the divider; CRLF with padded dividers; dividers before the first and after the last paragraph.
• test_counting_composition_negative (9 cases): *** with the right counts; *** with one-word sentences (main gives reward 1); one-word sentences; three-word sentences; three sentences; two and four paragraphs; an inline * * *; an empty paragraph between dividers.
• test_only_counting_composition_checker_is_replaced: same keys as the upstream registry, and only this entry differs.

With the new tests and main's app.py: 7 failed, 23 passed (the 5 positive cases get reward 0, the *** one-word case gets reward 1, and the registry test cannot import INSTRUCTION_DICT). With this change: 30 passed. The three examples from the issue now return True, False, False (main: False, False, True).

Rollouts

No model rollout was run. The change only affects how one checker scores a given text; the agent, prompts, data and other checkers are unchanged. Instead, every one of the 2,131 rows with this constraint was replayed through verify() (published row format, grading_mode="fraction") with three constructed responses built from that row's own n_sent and n_words. Counts are rows where the count:counting_composition entry of follow_instruction_list is true:

Response	main	this PR
* * * dividers, exact counts (follows the instruction)	0 / 2131	2131 / 2131
*** dividers, exact counts	0 / 2131	0 / 2131
*** dividers, one word fewer per sentence	2131 / 2131	0 / 2131

Happy to add a smoke rollout on some of these rows if you want model evidence.

Compatibility and benchmark impact

• Only rows that include count:counting_composition change: 2,131 of 46,391 (about 4.6%) in nvidia/Nemotron-RL-instruction_following. On those rows a compliant response can now pass this constraint, and the *** plus short sentences shape that passed before now fails, so rewards and metrics for them are not comparable with earlier runs. The reward profiling numbers in the README were measured with the old checker and were not re-run.
• No config, API or dependency change. INSTRUCTION_DICT is a new module-level name in app.py.
• benchmarks/ifeval uses this server, but the IFEval prompts carry no count: instructions, so its scores do not change.

Related PRs

• feat(instruction_following): add loose-mode evaluation and compute_metrics #1720 (loose-mode evaluation, same server): the two PRs change different lines. feat(instruction_following): add loose-mode evaluation and compute_metrics #1720 already conflicts with main in app.py and tests/test_app.py; merging this branch adds no new conflict hunks there. Its loose check reuses the instruction instance built for the strict check, so after both land loose mode uses the fixed checker too. Its fallback calls check_following(text, mode="loose") and catches TypeError; the override keeps upstream's one-argument signature, so it takes the same perturbation path as every other checker.
• feat(indicifeval): add IndicIFEval Trans benchmark #3695 (IndicIFEval backend): adds an import at the same place in app.py, so whichever merges second keeps both import lines. No other overlap.
• feat(checkpoint): declare twenty training verifiers stateless with replayable verification #3887 (stateless checkpoint flags): merges cleanly with this branch.

Proposed labels: bug, area:environment, community-request.

Checklist

• I have read the contributing guidelines.
• The change is focused; unrelated "drive-by" edits are tracked as separate issues/PRs.
• Tests added or updated and pass locally, or N/A for docs-only / non-code changes (so CI unit/server checks pass when applicable).
• Documentation added or updated, or N/A with justification.
• Pre-commit checks pass locally (pre-commit run --all-files) (so CI lint/format checks pass).
• New source files include the required Apache-2.0 SPDX header.
• All commits have DCO sign-off (git commit -s) (so the DCO check passes).

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[marcromeyn]

/ok to test 5ba0c55

[marcromeyn]

/ok to test 70485f9

[cmunley1]

/review

[svcnvidia-nemo-ci]

HOLD — the required model smoke validation is missing (RISK).

RISK — Validate the changed reward behavior with real model responses

Failure and scope: The English verify() path now grades count:counting_composition with a different checker. The PR explicitly reports no model rollout; its validation uses constructed responses, including the new positive fixtures. Trusted-base AGENTS.md requires representative real model smoke rollouts and inspection for behavior-changing environment work. Constructed-response replay is useful but does not satisfy that requirement. This is a validation gap, not a demonstrated checker failure; it concerns English tasks carrying this constraint.

Correction: Run a small smoke using the existing instruction-following configuration on affected published rows with varied n_sent/n_words, including a row with another constraint. Inspect the generated answers and record the row arguments, checker decisions, and rewards in the PR validation evidence. No checker rewrite is requested unless that inspection exposes a mismatch. The invariant is that real model outputs receive scores consistent with the advertised constraints.

Acceptance and preservation — proposed, not executed: Manually check each sampled answer’s divider, sentence, and tokenizer-based word counts against its follow_instruction_list entry; confirm binary reward equals float(all(bits)) and fractional reward equals sum(bits) / len(bits). Retain the existing positive cases’ reward 1.0/[True], negative cases’ reward 0.0/[False], and the registry-isolation assertions. In particular, spaced dividers with punctuation, CRLF, and outer dividers must keep passing, while contiguous *** and short sentences must keep failing.

All four changed files and the directly affected routing and reward paths were inspected. This was a read-only static review; test and replay results remain author-reported.

Note: formal approval is not available for this request, so this review is posted as a comment only.

Review generated by codex. This was a full review. Request a full review with /review reset=true.

[cmunley1]

tested a rollout, lgtm thanks!

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40054917034

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40059189099

[github-actions]

🔄 Merge queue validation started! You can track the progress here: https://github.com/NVIDIA-NeMo/Gym/actions/runs/40062258383