Georgia Tech · Reasoning and Learning Group
CreSTAL · In-Silico Materials-Targeting RL
CreSTAL extends LLMatDesign's crystal-editing workflow with supervised fine-tuning (SFT) and reinforcement learning (RL) from executable feedback. Programmatic checks, learned relaxation, and GNN band-gap predictions make proposed edits trainable. Evaluation asks whether successive editing attempts reach a fixed target.
Paper and appendix
CreSTAL: An In-Silico Reinforcement Learning Environment for Targeted Materials Editing was submitted to the AAAI-27 Student Abstract and Poster Program.
From proof of concept to study
I first used Qwen3-0.6B as a proof of concept to validate the end-to-end materials-editing and tool-calling loop. That model is outside the paper's benchmark. The reported study scaled the experiments to Qwen3-4B, Minitron-4B, Llama-3.2-3B, and Phi-4-mini.
Problem
Can learning from individual crystal edits improve property targeting over successive attempts? Models first learn from demonstrated edits, then receive one scalar RL reward per proposal. Evaluation allows up to seven attempts to reach a final recorded band gap within 1.4 ± 0.05 eV.
System
At each step, the model emits an add, remove, substitute, or exchange operation. Programmatic checks evaluate the format, operation, and chemistry, while a reward-relevant but non-fail-stop geometry check evaluates the candidate. A MatDeepLearn force field performs ASE/FIRE structure relaxation, and a GNN surrogate predicts the band gap. Those checks, outcomes, relaxation results, and diagnostics form a validation record from which one scalar reward is computed for GRPO training. Parsing, execution, or chemistry failures can stop a proposal before property prediction. Passing the relaxed charge check does not establish charge neutrality, and the separate stability check is disabled.
The model sees the formula, current band gap, and objective; coordinates and lattice vectors remain in the environment. During evaluation, a summary of up to five recent attempts is added to the prompt. Detailed failure reasons remain in the logs and are absent from this summary. SFT and single-edit RL inputs contain no previous attempts.
From proposal to recorded outcome
- Propose an edit from the current formula, gap, and target.
- Parse and apply the edit; check composition.
- If eligible, relax with a learned force field and predict the gap.
- Record checks, outcomes, and diagnostics.
Training: one edit
Validation record → one scalar reward per proposal. No attempt history.
Evaluation: up to seven attempts
Update state and recent-attempt summary → next proposal. Stop at an executed target hit or the attempt limit.
Evaluation protocol
The benchmark evaluates 1,000 materials drawn from an 8,680-material Materials Project pool, disjoint by material ID from training and balanced across four initial-gap strata. Each material receives one stochastic trajectory. The main study compares base, supervised fine-tuned, and GRPO-post-trained checkpoints under the same evaluator, target, edit budget, and success tolerance. Each primary RL run uses 1,000 training steps. Target-hit rate counts materials whose final recorded gap lies within 1.4 ± 0.05 eV. Execution rate counts attempted edits logged as successfully executed. The appendix also reports distance-based diagnostics.
Starting gaps come from the dataset; executed edits receive updated property predictions. If no edit executes, the final recorded gap remains the starting dataset value. A target hit therefore need not represent a newly achieved target.
| Model | SFT hits | RL hits | SFT execution | RL execution |
|---|---|---|---|---|
| Qwen3-4B | 111/1000 (11.1%) | 148/1000 (14.8%) | 64.1% | 55.3% |
| Minitron-4B | 121/1000 (12.1%) | 135/1000 (13.5%) | 65.8% | 59.9% |
| Llama-3.2-3B | 121/1000 (12.1%) | 119/1000 (11.9%) | 61.2% | 48.9% |
| Phi-4-mini | 118/1000 (11.8%) | 104/1000 (10.4%) | 59.8% | 74.7% |
All four base models recorded 12/1000 hits (1.2%) from unchanged materials already inside the band and executed no edits. RL columns use the original reward. Execution percentages use attempted edits as their denominator, not the 1,000 starting materials.
Result: mixed targeting gains across models
RL increased target hits for Qwen and Minitron and decreased them for Llama and Phi. Qwen rose from 11.1% after SFT to 14.8% with the original reward, while its execution rate fell from 64.1% to 55.3%. Executing more edits and reaching more targets are distinct outcomes.
Qwen's paired gain was 3.7 percentage points (exact McNemar p = 0.015). After Holm correction across the four model comparisons, p = 0.059, above the 0.05 significance threshold. These tests concern the evaluated materials and checkpoints; variation across repeated training runs remains unmeasured.
Target hits out of 1,000 materials
Bar scale: 0–200 hits. SFT in blue; original-reward RL in orange.
Qwen3-4B
Minitron-4B
Llama-3.2-3B
Phi-4-mini
I tested three reward formulations with Qwen3-4B: the original reward, Progress–Success with a strict-target bonus, and Tiered Target with progressively narrower target bonuses and reduced validity credit. Strict success was 14.8%, 15.6%, and 15.7%, respectively, compared with 11.1% after SFT. These were single adaptive runs, so the differences do not establish reward superiority.
My role
As a co-first author and researcher, I built and maintained the end-to-end SFT/GRPO post-training pipeline, materials-editing environment integration, data-generation pipeline, and evaluation and regression checks. I also orchestrated distributed training and rollout generation across four H100 workers, worked through tokenizer, log-probability, and GPU-utilization failures, and contributed to the manuscript's analysis. The linked writing captures several of those debugging investigations; the benchmark results represent the broader team's work.
Limitations
- Each model–reward setting has one training run and one stochastic evaluation trajectory per material.
- Post-edit band gaps use the same learned predictor for reward information and target-hit assessment; unchanged materials retain their starting dataset values.
- Edited structures were not independently evaluated with DFT or experiment, and the environment does not certify physical materials.
- GRPO optimizes individual edits while evaluation allows up to seven attempts.
- The Qwen reward follow-ups are adaptive and non-confirmatory; only one band-gap target is evaluated.