Why Does My GPU Utilization Keep Dipping?

This post details one of many frustrations I experienced this past year in building out my research project, CreSTAL. Currently, we are writing the research paper, so here is a debug that serves as a complement. For transparency: this one is doubly AI-assisted — the debugging itself was a conversation with an agent that had my repo checked out, and I used Claude Fable to synthesize that conversation and my GPU telemetry into this concise breakdown. Enjoy!

TL;DR: My colocated GRPO runs produce a sawtooth GPU-utilization trace: ~90% for a stretch, then sharp dips to 13–40%, over and over, plus occasional spikes in time-spent-accessing-memory. I confidently diagnosed a “periodic data-transfer bottleneck” — vLLM handing tensors off to my GNN reward workers. Then I asked a coding agent with repo access to verify, and it returned “partially correct,” which is the polite AI way of saying the graph is real but your mechanism is fiction. The dips are GRPO’s phase cycle made visible by colocation: vLLM decode, then CPU parsing plus GNN reward scoring, then a cross-rank gather that waits on the slowest straggler, then the HF forward/backward and weight sync back into vLLM. Nothing hands off tensors — vLLM outputs text. When everything shares one GPU, your utilization trace is just your training loop’s anatomy.

The setup

Same pipeline as the last two posts, but this time nothing is broken — I was just finally looking at the hardware telemetry. One node, four H100s. Each rank runs three tenants on its GPU: the GRPO trainer with the DDP policy model, a colocated vLLM engine capped at 30% GPU memory (TRL’s colocate mode), and two “modification workers” — GNN surrogates that execute each proposed materials modification and score the resulting property. 32 completions per prompt.

Three graphs, one story to explain:

• GPU memory allocated: dead flat at ~73%. Three tenants, one GPU, everyone pre-allocated. Boring — which is what you want from a memory chart.
• GPU utilization: a sawtooth. Long stretches pinned at 85–91%, then sharp dips to 13–40%, recurring every one to four minutes, with the dips thinning out later in the run.
• GPU time spent accessing memory: near zero almost always, with isolated spikes to ~60% roughly every seven minutes.

My confident hypothesis

Here’s what I wrote at the time, verbatim:

What you are seeing is a periodic data-transfer bottleneck. Most of the time, vLLM is happily generating tokens at 90% utilization. But periodically — likely when vLLM hands off data to the GNN workers, or when the GNN workers run their property improvements and synchronize — a large amount of data has to be moved in memory. During these split seconds, the GPU gets bogged down moving memory around, which forces the compute cores to pause and wait.

It has everything a good performance story needs: a villain (data movement), a mechanism (the handoff), and a perfect fit to the graphs. So instead of shipping it to my notes as fact, I asked an agent with the repo checked out to trace the actual step and tell me if I was right.

The verdict: “partially correct”

The periodicity was real. The dips were real. Nearly every mechanism I named was wrong. What a GRPO step in this pipeline actually does is a sequential four-phase loop, and the utilization trace is just those phases taking turns:

Phase 1 — decode (the ~90% plateau). vLLM chews through a 32-completion batch. Autoregressive decode is compute-heavy and long-running; utilization pins high. This is most of the wall clock, which is why the trace is mostly plateau.

Phase 2 — reward scoring (the dips begin). The completions leave vLLM as text. Not tensors — text. The reward function parses the XML on CPU, converts structures to ASE atoms, and ships them to the modification workers over a process pool with pickle IPC. The GNN property predictions do run on the same GPU, but they’re small, bursty models that don’t saturate the SMs. Utilization dips not because the GPU is “bogged down moving memory,” but because the work changed shape: from a dense decode batch to CPU parsing punctuated by lightweight GNN inference.

Phase 3 — the gather (the dips deepen). Four ranks reach a collective gather of rewards, and every rank waits for whichever GNN job is slowest anywhere on the node. That’s idle-by-synchronization: three GPUs politely doing nothing because a fourth is still relaxing a crystal structure.

Phase 4 — train and sync. The HF policy runs forward/backward, the LoRA update lands, and TRL syncs the updated weights back into the colocated vLLM engine so the next batch is generated by the current policy. This is the one place in the loop where a genuinely large memory move happens on schedule — the closest thing to my “data transfer” villain, except it’s weight traffic, not structure handoff.

The scorecard on my original story:

My claim	Verdict
Periodic bottleneck tied to the step cycle	Yes — generate → reward → gather → train, repeating
vLLM at ~90% during generation	Yes
Dips when generation ends	Yes — reward scoring, gather stragglers, training
”vLLM hands off data to GNN workers”	No — vLLM outputs text; the pool gets ASE structures via CPU pickle
”GNN workers synchronize” causes dips	Partially — the workers are async pool jobs; the real sync is the cross-rank gather
”Large memory moves force compute to pause”	Partially — real memory pressure exists (three tenants, 73% allocated), but the structure IPC is CPU-side; weight sync is the better memory story

The corrected mental model, as one picture:

And the memory-access spikes? Honest answer: still unconfirmed. Weight sync and cache management are the leading suspects, but the cadence (~7 minutes) doesn’t cleanly match either, and I refuse to get fooled by a plausible story twice in one debugging session. The correct move — the one the whole post is about — is to correlate the spikes against step boundaries in the training logs before believing anything.

How to check your own sawtooth

If your colocated GRPO trace looks like mine, four checks turn vibes into facts:

1. Align the dips with step boundaries. Plot global_step or step_time from your training logs against the utilization trace. If dips land on phase transitions, it’s anatomy, not pathology.
2. Break down time per phase. TRL already wraps generation, reward computation, and training in profiling contexts — compare where the seconds actually go before optimizing anything.
3. Correlate memory spikes with named events. Weight syncs, checkpoint saves, cache clears — each has a schedule. A spike that doesn’t match any schedule is the only one worth chasing.
4. Check dip depth against the slowest worker. On multi-GPU runs, if deeper dips track the longest reward job in the batch, your cost center is the straggler, not the GPU.

Takeaways

Colocation turns your utilization trace into an anatomy lesson. Trainer, vLLM, and reward workers take turns on the same silicon, so every phase transition is visible as a dip. That’s not a defect — it’s legibility. The question is never “why isn’t it pinned at 100%,” it’s “which phase is the longest, and is that the one I’m paying for?”

A utilization dip is not automatically a bottleneck. vLLM idling while rewards compute is the design of the loop. The optimization target is phase duration (faster reward scoring, fewer stragglers), not dip elimination.

Plausible is not correct. My story fit every pixel of the graphs and misidentified the mechanism at every step — the payload (text, not tensors), the location (CPU pickle, not GPU memory), and the synchronization (cross-rank gather, not worker handoff). Graphs cannot arbitrate between stories that fit them equally well. Only the code can.

“Partially correct” is a gift — take it. The agent kept what the graphs actually supported (real periodicity, real dips, real memory pressure) and relabeled the mechanisms with file-and-line receipts. That’s the same discipline as the last two posts pointed at a softer target: not a broken run this time, just a broken explanation. Those are cheaper to fix before they end up in a paper.

References

• TRL docs — vLLM integration: colocate mode and weight syncing
• Why is my GRPO Loss 0? — same pipeline, when the run actually was broken
• Why Is My Model Suddenly Speaking Thai? — same pipeline, when the tokenizer was the villain