skip to content
Jordan Madison
Table of Contents

I hit this bug while building CreSTAL. As we write the research paper, I’m publishing some of the debugging work that sits behind it. I used Claude Fable to turn my notes, project code, and screenshots into this account.

TL;DR: I added custom XML tags (<hypothesis>, <modification>, …) to a tokenizer as special tokens, SFT’d a model to use them, and everything looked perfect in evaluation. The moment the same model entered a GRPO rollout, the tags vanished and the model started emitting random Thai and CJK tokens where the tags should be.

Two separate failures produced similar-looking output. skip_special_tokens=True defaults across TRL and vLLM could erase correctly sampled tags before my parser saw them. But the Thai and CJK substitutions had a different cause: my LoRA run had left the new embedding and LM-head rows frozen, so the tag IDs were never learned well enough to be sampled. Tokenizer setup had also drifted across the SFT, eval, merge, and RL paths, which made both failures harder to isolate. I fixed the pipeline by adding the tags as regular tokens, training and saving their rows, and moving all tokenizer setup into one shared module.

The setup

I’m post-training small open models (Qwen3-4B, Llama 3.2 3B, Phi-4-mini, Minitron-4B) to propose materials modifications: “substitute nickel with nitrogen to raise the band gap,” that kind of thing. The pipeline is SFT to teach the output format, then GRPO to optimize against a reward.

The output format is structured XML so a downstream parser can extract the proposal:

<hypothesis>Substituting Ni with N in Ba(NiP2)2 is expected to
increase the band gap because...</hypothesis>
<modification>["substitute", "Ni", "N"]</modification>

Those tags aren’t in any base vocabulary, so I added them to the tokenizer the way most tutorials suggest:

NEW_TOKENS = ["<hypothesis>", "</hypothesis>", "<modification>",
              "</modification>", "<previous_attempts>", "</previous_attempts>"]
tokenizer.add_tokens(NEW_TOKENS, special_tokens=True)
model.resize_token_embeddings(len(tokenizer))

SFT went fine. My evaluation script showed the fine-tuned Qwen producing exactly the right structure, greedy-decoded, tags and all:

<hypothesis>Substituting thulium (Tm) with fluorine (F) in TmMgHg2
could increase the band gap...</hypothesis><modification>["substitute", "Tm", "F"]</modification>

Ready for RL.

The symptom

Same model. Same prompt. Same (I believed) tokenizer. But inside the GRPO rollout, completions came back looking like this:

แบ่งThe substitution of Ni with N in Ba(NiP2)2 to form BaN3P2 is
expected to increase the band gap due to several structural and
chemical factors... เปล่าร่างกาย["substitute", "Ni", "N"]สมเด็<|im_end|>

The content was intact (a coherent hypothesis, a well-formed JSON list), but every place a tag should appear, there was a random Thai (sometimes Chinese, sometimes Arabic) token instead. แบ่ง where <hypothesis> belonged. เปล่าร่างกาย where </hypothesis><modification> belonged. The model clearly knew the format. It just couldn’t say the tags.

Because the parser found no <hypothesis> block, every completion failed format checking, every reward came back 0, and the run learned nothing:

'rewards/reward_fn/mean': 0.0, 'reward_std': 0.0, 'frac_reward_zero_std': 1.0

If you read my last post, you know where identical rewards lead in GRPO: every advantage in the group is zero, and the run burns GPU hours without learning anything.

The goose chase

I’ll spare you the two days of Slack messages and just list what I tried, roughly in order of increasing desperation:

  • Set skip_special_tokens=False in the TRL/GRPOTrainer generation kwargs
  • Temperature sweeps across RL runs (0.0, 0.3, 0.6, 0.7); it wasn’t sampling noise, since greedy decoding produced the same garbage
  • Verified the tokenizer loaded into the GRPOTrainer was the same one used in evaluation, and that its added_tokens.json contained all six tags with the IDs I expected (151669–151674 on Qwen3)
  • Read through TRL’s grpo_trainer.py generation path and flipped skip_special_tokens in _generate() and _generate_and_score_completions()
  • Flipped it again in chat_template_utils.parse_response()
  • Passed skip_special_tokens=False into vLLM’s SamplingParams (it defaults to True there too), only to find the code path I was worried about never branched there
  • Re-ran SFT with more epochs (3 → 10) in case the format just hadn’t been learned hard enough
  • Learning-rate sweeps, in case early GRPO updates were destroying the format

None of it changed anything. Time to stop guessing and look at token IDs.

Looking at the actual IDs

I added print statements everywhere completion IDs are produced and decoded, and re-ran a debug job. This was the moment the problem changed shape:

Completion 0: len=163, has_special_id=False, last_5_ids=[330, 45, 1341, 141044, 151645]
Completion 1: len=150, has_special_id=False, last_5_ids=[330, 40, 1341, 146185, 151645]
Completion 2: len=98,  has_special_id=False, last_5_ids=[330, 22571, 1341, 140332, 151645]
[151672] '</modification>'

Line those IDs up against the decoded text and the pattern jumps out. Every completion ends the same way: the tail of the JSON list (..."N"], with ID 1341 recurring in the second-to-last slot across completions), then one random high-ID token, then 151645, which is <|im_end|>. The closing </modification> tag is ID 151672. The tokens the model actually sampled in its place (141044, 146185, 140332, a different one every time) are all from the same far-end neighborhood of the vocabulary, which for Qwen is where the Thai, CJK, and other multilingual tokens live.

So this wasn’t only a decoding-display problem. has_special_id=False on every single completion: the tag IDs were never sampled at all. Instead, the model sampled unrelated high-ID multilingual tokens at exactly the positions where the tags belonged. Decode flags can hide a sampled token, but they cannot replace its ID with a different one.

That reframing mattered, because it meant I had two distinct problems producing one symptom.

Problem 1: skip_special_tokens=True is everywhere

When you call tokenizer.add_tokens(new_tokens, special_tokens=True), you reserve vocabulary slots and opt those tokens into every piece of special-token machinery in the stack. The most dangerous piece is decoding. skip_special_tokens=True removes special tokens from decoded text, and it is the default (sometimes hard-coded) in more places than you’d expect:

  • TRL’s GRPOTrainer decodes completions with skip_special_tokens=True before your reward functions see them (trl#2897, trl#3026)
  • vLLM’s SamplingParams defaults skip_special_tokens=True for the text it returns
  • Any tokenizer.decode() call in your own parsing utilities that passes the flag through

The failure mode is nasty precisely because nothing errors. Your model can emit perfectly formatted output, and the string handed to your format-checking reward function simply won’t contain the tags. Every reward is 0, every group has zero variance, every advantage is zero, and GRPO trains on nothing. My evaluation script never hit this because I controlled the decode call there and passed skip_special_tokens=False myself.

This Hugging Face forum answer explains what “special” means across slow tokenizers, fast tokenizers, and inference engines. I wish I’d read it on day one. The short version: added_tokens.json is “vocabulary added on top of the base vocab,” and the special flag is a low-level switch controlling how the fast tokenizer treats those tokens during pre-tokenization and decode. It is not just metadata.

Problem 2: the tag rows never trained

Flipping decode flags explained missing tags in text, but not missing tag IDs in the sampled tokens. The training-side failure was in my LoRA config: it adapted the attention and MLP projections while leaving both the resized embedding layer and LM head frozen. The new tag rows stayed near their initialization instead of learning distinct input and output representations. Part 2 shows the embedding statistics and the modules_to_save fix.

A separate engineering problem made this harder to diagnose. Over weeks of iteration, token addition, embedding resizing, and tokenizer loading had been implemented slightly differently in the SFT script, eval script, RL pipeline, and adapter-merge step. That drift meant I could not initially tell whether vLLM had received the same tokenizer and trained rows I had evaluated.

The final fix covered both failure modes. I moved every mention of token addition, embedding resizing, and tokenizer setup into one module that SFT, eval, and RL all import. It adds the CreSTAL tags as regular tokens (special_tokens=False), resizes the model in exactly one place, and ensures the embedding layer and LM head are trained and saved with the adapter. Regular added tokens still tokenize as single indivisible pieces, but no skip_special_tokens flag can erase them. After retraining, the GRPO rollouts produced the same clean tagged output as the evaluation script.

Takeaways

  • Don’t make your format tags special tokens unless you have a reason to. “Special” protects them from the pre-tokenizer, but regular added tokens do too. It also lets default decode calls silently delete them before they reach your reward function. If your reward depends on seeing a token, the safest move is for that token not to be special.

  • Use one tokenizer setup path, period. SFT, eval, and RL must construct the tokenizer and resize embeddings through the same function. The moment that logic is copy-pasted, the copies will drift, and tokenizer drift produces exactly this kind of “works here, gibberish there” symptom.

  • When text looks wrong, look at IDs. I lost days reasoning about decoded strings. One print of last_5_ids next to the expected tag ID turned “the model is degrading during RL” into “the tag IDs are never sampled,” a different and much more tractable bug.

  • Watch frac_reward_zero_std. When it’s 1.0, your reward function is returning the same value for every completion in every group, and GRPO is a no-op. It’s the fastest sign that something upstream of the reward is broken: your tags being deleted, for instance.

  • Search your entire stack for skip_special_tokens. Transformers, TRL, vLLM, and your own utilities all have this flag, with different defaults, some hard-coded. Know where every one of them is before you train.

References