skip to content
Jordan Madison
Table of Contents

This post details one of many frustrations I experienced this past year in building out my research project, CreSTAL. Currently, we are writing the research paper, so here is a debug that serves as a complement. For transparency: I used Claude Fable to synthesize my written notes, project code, and results screenshots into this concise breakdown. Enjoy!

TL;DR: I added custom XML tags (<hypothesis>, <modification>, …) to a tokenizer as special tokens, SFT’d a model to use them, and everything looked perfect in evaluation. The moment the same model entered a GRPO rollout, the tags vanished and the model started emitting random Thai and CJK tokens where the tags should be. Two things were going on: skip_special_tokens=True defaults are scattered across TRL and vLLM and silently erase special tokens from the text your parsers and reward functions see, and my tokenizer setup logic had drifted across the SFT, eval, and RL paths. The fix: add the tags as regular tokens instead of special ones, and consolidate all tokenizer setup into a single shared module. If a token matters to your reward function, either don’t make it special, or hunt down every decode call in the stack.

The setup

I’m post-training small open models (Qwen3-4B, Llama 3.2 3B, Phi-4-mini, Minitron-4B) to propose materials modifications — “substitute nickel with nitrogen to raise the band gap,” that kind of thing. The pipeline is SFT to teach the output format, then GRPO to optimize against a reward.

The output format is structured XML so a downstream parser can extract the proposal:

<hypothesis>Substituting Ni with N in Ba(NiP2)2 is expected to
increase the band gap because...</hypothesis>
<modification>["substitute", "Ni", "N"]</modification>

Those tags aren’t in any base vocabulary, so I added them to the tokenizer the way most tutorials suggest:

NEW_TOKENS = ["<hypothesis>", "</hypothesis>", "<modification>",
              "</modification>", "<previous_attempts>", "</previous_attempts>"]
tokenizer.add_tokens(NEW_TOKENS, special_tokens=True)
model.resize_token_embeddings(len(tokenizer))

SFT went fine. My evaluation script showed the fine-tuned Qwen producing exactly the right structure, greedy-decoded, tags and all:

<hypothesis>Substituting thulium (Tm) with fluorine (F) in TmMgHg2
could increase the band gap...</hypothesis><modification>["substitute", "Tm", "F"]</modification>

Ready for RL. Or so I thought.

The symptom

Same model. Same prompt. Same (I believed) tokenizer. But inside the GRPO rollout, completions came back looking like this:

แบ่งThe substitution of Ni with N in Ba(NiP2)2 to form BaN3P2 is
expected to increase the band gap due to several structural and
chemical factors... เปล่าร่างกาย["substitute", "Ni", "N"]สมเด็<|im_end|>

The content was intact — a coherent hypothesis, a well-formed JSON list — but every place a tag should appear, there was a random Thai (sometimes Chinese, sometimes Arabic) token instead. แบ่ง where <hypothesis> belonged. เปล่าร่างกาย where </hypothesis><modification> belonged. The model clearly knew the format. It just couldn’t say the tags.

And because the parser found no <hypothesis> block, every completion failed format checking, every reward came back 0, and the run learned nothing:

'rewards/reward_fn/mean': 0.0, 'reward_std': 0.0, 'frac_reward_zero_std': 1.0

If you read my last post, you know where identical rewards lead in GRPO: every advantage in the group is zero, and the trainer politely does nothing while burning GPU hours.

The goose chase

I’ll spare you the two days of Slack messages and just list what I tried, roughly in order of increasing desperation:

  • Set skip_special_tokens=False in the TRL/GRPOTrainer generation kwargs
  • Temperature sweeps across RL runs (0.0, 0.3, 0.6, 0.7) — it wasn’t sampling noise; greedy decoding produced the same garbage
  • Verified the tokenizer loaded into the GRPOTrainer was the same one used in evaluation, and that its added_tokens.json contained all six tags with the IDs I expected (151669–151674 on Qwen3)
  • Read through TRL’s grpo_trainer.py generation path and flipped skip_special_tokens in _generate() and _generate_and_score_completions()
  • Flipped it again in chat_template_utils.parse_response()
  • Passed skip_special_tokens=False into vLLM’s SamplingParams (it defaults to True there too), only to find the code path I was worried about never branched there
  • Re-ran SFT with more epochs (3 → 10) in case the format just hadn’t been learned hard enough
  • Learning-rate sweeps, in case early GRPO updates were destroying the format

None of it moved the needle. Time to stop guessing and look at token IDs.

Looking at the actual IDs

I added print statements everywhere completion IDs are produced and decoded, and re-ran a debug job. This was the moment the problem changed shape:

Completion 0: len=163, has_special_id=False, last_5_ids=[330, 45, 1341, 141044, 151645]
Completion 1: len=150, has_special_id=False, last_5_ids=[330, 40, 1341, 146185, 151645]
Completion 2: len=98,  has_special_id=False, last_5_ids=[330, 22571, 1341, 140332, 151645]
[151672] '</modification>'

Line those IDs up against the decoded text and the pattern jumps out. Every completion ends the same way: the tail of the JSON list (..."N"] — note ID 1341 recurring in the second-to-last slot across completions), then one random high-ID token, then 151645, which is <|im_end|>. The closing </modification> tag is ID 151672. The tokens the model actually sampled in its place — 141044, 146185, 140332, a different one every time — are all from the same far-end neighborhood of the vocabulary, which for Qwen is where the Thai, CJK, and other multilingual tokens live.

So this wasn’t a decoding-display problem. has_special_id=False on every single completion: the tag IDs were never sampled at all. The model was reaching for the right region of the vocabulary at exactly the right positions — and grabbing a random neighbor every time. That’s not a model that forgot its training; that’s a model whose learned tag embeddings don’t exist in the weights being served.

That reframing mattered, because it meant there were two distinct problems wearing the same trench coat.

Problem 1: skip_special_tokens=True is everywhere

When you call tokenizer.add_tokens(new_tokens, special_tokens=True), you’re not just reserving vocabulary slots. You’re opting those tokens into every piece of special-token machinery in the stack — and the most dangerous piece is decoding. skip_special_tokens=True removes special tokens from decoded text, and it is the default (sometimes hard-coded) in more places than you’d expect:

  • TRL’s GRPOTrainer decodes completions with skip_special_tokens=True before your reward functions see them (trl#2897, trl#3026)
  • vLLM’s SamplingParams defaults skip_special_tokens=True for the text it returns
  • Any tokenizer.decode() call in your own parsing utilities that passes the flag through

The failure mode is nasty precisely because nothing errors. Your model can emit perfectly formatted output, and the string handed to your format-checking reward function simply won’t contain the tags. Every reward is 0, every group has zero variance, every advantage is zero, and GRPO trains on nothing. My evaluation script never hit this because I controlled the decode call there and passed skip_special_tokens=False myself.

There’s a genuinely good explainer on what “special” actually means across the slow tokenizer, the fast tokenizer, and inference engines in this Hugging Face forum answer — I wish I’d read it on day one. The short version: added_tokens.json is “vocabulary added on top of the base vocab,” and the special flag is a low-level switch controlling how the fast tokenizer treats those tokens during pre-tokenization and decode. It is not just metadata.

Problem 2: my tokenizer setup had drifted

Flipping decode flags explained missing tags in text, but not missing tag IDs in the sampled tokens. That second problem was self-inflicted. Over weeks of iteration, token addition, embedding resizing, and tokenizer loading had been reimplemented slightly differently in the SFT script, the eval script, and the RL pipeline — plus the adapter-merge step that produces the model vLLM actually serves. Any disagreement between those paths (tokens added in a different way, embeddings resized against a different checkpoint, one path adding them as special and another not) means the merged model vLLM serves doesn’t have the trained embedding rows where the tokenizer says the tags live. The model’s hidden states point at the right corner of embedding space, and inference samples whatever untrained neighbor is closest. Hence: Thai.

The fix was boring and effective. I moved every mention of token addition, embedding resizing, and tokenizer setup into one module that SFT, eval, and RL all import. It adds the CreSTAL tags as regular tokens (special_tokens=False), initializes the new embedding rows to the mean of the existing embeddings, and resizes the model in exactly one place. Regular tokens still tokenize as single indivisible pieces, but no skip_special_tokens flag anywhere in HF, TRL, or vLLM can ever erase them — which removes an entire category of silent failure from the pipeline. Fresh Python environment for good measure, re-ran SFT, and the GRPO rollouts finally produced the same clean tagged output the eval script had been showing all along.

Takeaways

Don’t make your format tags special tokens unless you have a reason to. “Special” buys you protection from the pre-tokenizer splitting them — but regular added tokens get that too. What “special” additionally buys you is being silently deleted by every default decode call between your model and your reward function. If your reward depends on seeing a token, the safest move is for that token not to be special.

One tokenizer setup path, period. SFT, eval, and RL must construct the tokenizer and resize embeddings through the same function. The moment that logic is copy-pasted, the copies will drift, and tokenizer drift produces exactly this kind of “works here, gibberish there” symptom.

When text looks wrong, look at IDs. I lost days reasoning about decoded strings. One print of last_5_ids next to the expected tag ID turned “the model is degrading during RL” into “the tag IDs are never sampled,” which is a completely different and much more tractable bug.

Watch frac_reward_zero_std. When it’s 1.0, your reward function is returning the same value for every completion in every group, and GRPO is a no-op. It’s the single fastest tell that something upstream of the reward — like, say, your tags being deleted — is broken.

grep -rn skip_special_tokens your entire stack. Transformers, TRL, vLLM, and your own utilities all have this flag, with different defaults, some hard-coded. Know where every one of them is before you train.

References