CreSTAL · In-Silico Materials-Targeting RL

From proof of concept to study

I first used Qwen3-0.6B as a proof of concept to validate the end-to-end materials-editing and tool-calling loop. That model is outside the paper's benchmark. The reported study scaled the experiments to Qwen3-4B, Minitron-4B, Llama-3.2-3B, and Phi-4-mini.

Problem

Executable rewards can make a language-model policy easier to optimize without improving the strict task that matters. CreSTAL studies that gap in an in-silico materials-editing setting. The task is to move each starting material's predicted band gap to 1.4 eV, within ±0.05 eV, using up to seven sequential symbolic edits.

System

At each step, the model emits an add, remove, substitute, or exchange operation. Programmatic checks evaluate the format, operation, and chemistry, while a reward-relevant but non-fail-stop geometry check evaluates the candidate. A MatDeepLearn force field performs ASE/FIRE structure relaxation, and a GNN surrogate predicts the band gap. Those checks and property signals become feedback for GRPO training. The resulting certificate is relative to the implemented surrogate environment, not a DFT, experimental, or physical-materials certificate.

Evaluation protocol

The benchmark evaluates 1,000 materials drawn from an 8,680-material Materials Project pool, disjoint by material ID from training and balanced across four initial-gap strata. Each material receives one stochastic trajectory. The main study compares base, supervised fine-tuned, and GRPO-post-trained checkpoints under the same evaluator, target, edit budget, and success tolerance. It measures three different things: continuous surrogate utility and progress, near-target landing within ±0.10/±0.20 eV, and strict success within ±0.05 eV.

* Base hits are the same 12 materials that began inside the target band; base models executed zero successful modifications. Qwen SFT has 996 evaluated materials.

Result: reward and task success can diverge

Held-out smooth property utility increased after RL for all four models, but strict target success varied by model. Qwen rose from 11.1% after SFT to 14.8% with the original reward; Minitron rose from 12.1% to 13.5%, while Llama and Phi declined slightly. This identifies a mismatch between reward improvement and strict task success under this protocol, not evidence that one model or reward is generally superior.

I tested three reward formulations with Qwen3-4B: Smooth Progress–Proximity, Progress–Success with a strict-target bonus, and Tiered Target with progressively narrower target bonuses and reduced validity credit. Strict success was 14.8%, 15.6%, and 15.7%, respectively, compared with 11.1% after SFT. These were single adaptive runs, so the differences do not establish reward superiority.

My role

As a co-first author and researcher, I built and maintained the end-to-end SFT/GRPO post-training pipeline, materials-editing environment integration, data-generation pipeline, and evaluation and regression checks. I also orchestrated distributed training and rollout generation across four H100 workers, worked through tokenizer, log-probability, and GPU-utilization failures, and contributed to the manuscript's analysis. The linked writing captures several of those debugging investigations; the benchmark results represent the broader team's work.

Limitations

• The study uses one training run and one stochastic trajectory per material.
• The learned surrogate both supplies reward information and defines success; its reported MAE of approximately 0.28 eV exceeds the ±0.05 eV target band.
• Edited structures were not independently evaluated with DFT or experiment, and the environment does not certify physical materials.
• GRPO optimizes single-step edits while evaluation composes up to seven edits.
• The Qwen reward follow-ups are adaptive and non-confirmatory; only one band-gap target is evaluated.

Code

CreSTAL on GitHub

Related writing

• Why Is My Model Suddenly Speaking Thai?
• When the GRPO Loss Is Exactly Zero
• The Embeddings Were Frozen
• Why GPU Utilization Dips During RL