skip to content
Jordan Madison
Table of Contents

This post details one of many frustrations I experienced this past year in building out my research project, CreSTAL. Currently, we are writing the research paper, so here is a debug that serves as a complement. For transparency: I used Claude Fable to synthesize my run logs, training configs, and W&B metrics into this concise breakdown. Enjoy!

TL;DR: My GRPO run logged a training loss of exactly 0.0 at every step — while rewards were healthy, reward variance was fine, and every completion parsed cleanly. The cause: vLLM and HuggingFace disagreed about the log-probabilities of the very tokens vLLM had just sampled — a ~0.3-nat-per-token, architecture-specific discrepancy that hit Minitron and Phi but not Qwen or Llama. Compounded over ~150-token completions, the sequence-level importance ratios collapsed to ~1e-10, and TRL’s default sequence_mask correction zeroed out every single sequence in every single batch. The model learned nothing, very stably. The fix: switch the correction to per-token truncation (vllm_importance_sampling_mode="token_truncate", cap 10) — clip the ratios instead of discarding the sequences.

The setup

Same project as always: post-training small open models to propose materials modifications. This run was Minitron-4B-Base, SFT’d to emit <hypothesis>/<modification> XML, then GRPO against a physics-based reward — each proposed modification gets executed and scored against the target band gap. TRL 0.28 with colocated vLLM 0.11 on 4×H100s, 32 completions per prompt, 1,000 steps planned.

Generation was configured like this:

generation_kwargs = {
    "temperature": 0.6,
    "top_p": 0.9,
    "top_k": 50,
    "repetition_penalty": 1.15,
    "skip_special_tokens": False,
    "stop": ["</modification>"],
    "stop_token_ids": [3, 256003],
    "max_tokens": 1024,
    "include_stop_str_in_output": True,
}

Keep an eye on that block — I spent a long time glaring at it as the prime suspect, and the stop strings come back later. Spoiler: it’s innocent.

The symptom

The run looked good. Completions were well-formed (parse_failure_rate: 0.0 across the entire run — the tag-vanishing saga was long behind me), rewards were varied, entropy was stable. And the training loss was zero. Not small. Not noisy around zero. Exactly 0.0, every logged step:

step   loss  grad_norm   reward  reward_std   IS_ratio_mean
  10    0.0    0.00530    0.819       0.683        3.97e-10
  20    0.0    0.00483    1.007       0.513        4.87e-09
  30    0.0    0.00628    0.605       0.602        2.01e-10
  40    0.0    0.00624    1.171       0.239        1.10e-07
  50    0.0    0.00840    1.080       0.529        1.53e-07
  60    0.0    0.00736    0.925       0.494        1.28e-08
  70    0.0    0.00895    1.078       0.204        1.64e-10

Ignore the last column for now — I did too, for longer than I’d like to admit.

The usual suspect

Ask anyone (or any GitHub issue) why a GRPO loss is zero and you’ll get the same answer: your rewards are identical. GRPO normalizes advantages within each group of completions; if all 32 completions for a prompt score the same, every advantage is zero and the trainer politely does nothing while burning GPU hours. The tell is frac_reward_zero_std — the fraction of groups with zero reward variance.

So I checked. reward_std was healthy at every step (0.2–0.7). frac_reward_zero_std was 0.0 — not a single degenerate group. The logged completion tables showed nonzero advantages. The classic diagnosis simply didn’t apply: rewards varied, advantages existed, and the loss was still identically zero.

Which is unsettling in a specific way: the entire reward half of the pipeline — generation, parsing, physics scoring, group normalization — was provably working. The zero was being manufactured somewhere between the advantages and the gradient.

The tell

Two metrics I had never once looked at:

sampling/sampling_logp_difference/mean:    0.309
sampling/importance_sampling_ratio/mean:   3.97e-10
sampling/importance_sampling_ratio/max:    1.08e-08

That ratio is TRL comparing two opinions about the same tokens: the log-probabilities vLLM reported when it generated each completion, and the log-probabilities the training policy assigns when it scores them. In a healthy vLLM-backed GRPO run these are nearly the same model, so the ratio should hover around 1. Mine was ten orders of magnitude off. The per-token story is up top: a mean gap of 0.309 nats per token. That sounds small until you remember ratios multiply across the sequence — exp(-0.309 × 150) is about 1e-20. The logged sequence ratios ranged from 2e-23 to 1e-8. As far as the trainer could tell, the completions it was being asked to train on were essentially impossible under its own policy.

And TRL has machinery for exactly that situation. Because colocated vLLM generation is never perfectly in sync with the training policy, GRPOTrainer applies an importance-sampling correction to every sequence (docs) — and the default mode, sequence_mask, doesn’t downweight suspicious sequences, it zeroes them. Ratio outside the trust window → that sequence contributes nothing to the gradient. Every one of my sequences was at 1e-10. One hundred percent of every batch, masked. Loss: exactly 0.0. The tiny residual grad_norm of ~0.005 was the corpse twitching.

The safety mechanism worked flawlessly. What it was protecting me from turned out to be nothing I had written.

The cause

The prime suspect was that generation config. Any sampling knob vLLM applies that the trainer doesn’t replay when it scores completions — a temperature here, a repetition penalty there — shifts the ratio away from 1, and there’s a well-documented version of exactly that: by default vLLM reports logprobs before some of its sampling-parameter processing, so even a plain temperature mismatch can poison the correction (trl#4159, fixable with logprobs_mode="processed_logprobs").

But that theory had a hole in it. Every model family in this project trains through the same launcher with the same generation kwargs — and Qwen and Llama trained fine. The collapse only hit Phi-4-mini and Minitron: the two models in my lineup that aren’t a vanilla Llama-style architecture (Minitron is NVIDIA’s pruned Nemotron). If the config were guilty, everyone should have failed. When identical settings produce healthy ratios on two architectures and 1e-10 on the other two, the config isn’t the variable — the implementations are.

That’s what this was: a vLLM-versus-HuggingFace log-probability discrepancy specific to these architectures. The trainer computes its side of the ratio with the HF forward pass; vLLM reports its side from its own kernels; for the Nemotron and Phi architectures the two disagree by ~0.3 nats per token, systematically. The correction was faithfully measuring a gap between two implementations of the same checkpoint — and then dutifully masking every sequence the model produced.

The fix

Verbatim from the model config that shipped:

grpo:
  # Nemotron-arch model: guard against vLLM-vs-HF logprob mismatch (same failure
  # mode as Phi). token_truncate clips the per-token IS ratio instead of letting
  # the default sequence_mask (cap 3.0) zero out importance_sampling_ratio + loss.
  vllm_importance_sampling_mode: "token_truncate"
  vllm_importance_sampling_cap: 10.0

Same correction machinery, different philosophy. sequence_mask is all-or-nothing: a sequence whose cumulative ratio leaves the trust window contributes exactly zero gradient — and when the per-token gap is systematic, that’s every sequence, forever. token_truncate instead clips each per-token ratio at the cap: the update is biased, but it’s bounded, and it exists. Loss came back nonzero, gradients flowed, and the same two lines went into the Phi config. Or as the research paper puts it, in its Sunday best: “per-token importance-sampling truncation with a ratio cap of 10 for Phi and Minitron, correcting a vLLM–HuggingFace log-probability discrepancy that would otherwise zero the loss.”

Two honest caveats. Loosening the correction is a judgment call, not a free lunch — you’re choosing to train through a measured mismatch because you’ve convinced yourself it’s a systematic reporting artifact, not genuine off-policy drift. And if you can kill the drift at the source — sampling knobs in GRPOConfig fields where TRL replays them, logprobs_mode="processed_logprobs" on vLLM ≥ 0.10.2 — do that first, and let the correction go back to catching real problems.

Bonus red herring

While staring at these logs I also found completions/clipped_ratio: 1.0 — TRL claiming 100% of completions hit the max length without terminating, all run long. Alarming! Also false: every completion ended exactly at </modification>, because that’s my stop string. TRL only counts a completion as “terminated” if it ends with an EOS token it knows about; custom stop/stop_token_ids don’t qualify, so the bookkeeping calls them all truncated. Harmless here — but if I’d had mask_truncated_completions=True switched on, it would have zeroed every sequence a second, independent way. If your stop condition is a custom tag, this metric lies to you.

Takeaways

“Loss is 0” is a symptom class, not a diagnosis. I now know three distinct ways GRPO produces it: zero advantages (identical rewards in a group — check reward_std and frac_reward_zero_std), the vLLM importance-sampling correction masking everything (check sampling/importance_sampling_ratio), and truncation masking (check completions/clipped_ratio, then check whether it’s even telling the truth). Figure out which zero you have before changing anything.

The importance ratio should hover near 1. It’s the trainer telling you, every ten steps, how far your generation engine has drifted from your policy. 1e-10 isn’t drift; it’s two different distributions wearing the same checkpoint. I’d stared past that column for the entire run.

Same config + different architecture + different outcome = suspect the implementations. Identical launcher, identical kwargs; Qwen and Llama healthy, Phi and Minitron at 1e-10. The variable wasn’t anything I wrote — it was whose forward pass computed the logprobs. Models off the beaten architectural path earn extra skepticism at every vLLM↔HF boundary.

Masking discards; truncation degrades. sequence_mask is the right default when ratio blowups are occasional noise — but against a systematic mismatch it’s a silent off switch for training. token_truncate trades a bounded bias for an actual gradient. Know which failure you have before you pick.

grad_norm is the cheapest alarm you’re not watching. Healthy rewards plus a grad_norm of 0.005 means the gradient died somewhere between the reward function and the optimizer. That narrows the search enormously — I just wasn’t looking.

This run was stopped at step 76 of 1,000 — about 24 minutes on 4 GPUs. The vanishing-tags bug cost me two days. I’d like to think the trendline is the lesson sinking in: stop reasoning about what the trainer should be doing and go read what it logged.

References