Zero RL is the approach of training language models on verifiable task rewards with no human-labeled preference data at all. A model attempts a problem, its answer is checked programmatically, and the reward signal flows backward. No annotators, no preference rankings, no RLHF pipeline. DeepSeek’s R1 made this tractable at moderate scale; the question has been whether it holds as models get larger. Ring-Zero is an attempt to find out: the paper scales zero RL to 1 trillion parameters and documents what changes.
The resulting model is Ring-2.5-1T-Zero. Training at this scale introduces stability challenges that don’t appear at smaller sizes. The authors address three of them directly. Clipped importance sampling controls the distribution gap between the policy used to generate training rollouts and the policy being updated — a gap that widens at scale because large models shift more dramatically per gradient step. Training-inference ratio correction handles the mismatch between how many update steps the model takes per rollout versus how many inference steps it produces; at 1T parameters this ratio can drift in ways that destabilize the gradient signal. Mixed-precision control addresses the compounding errors in gradient accumulation that arise when you’re summing across trillions of parameters at reduced numerical precision.
The training dynamics show a two-phase pattern the authors call discovery and sharpening. In the discovery phase, the model is finding strategy templates — learning that certain reasoning patterns correlate with correct answers across the verifiable reward tasks. In the sharpening phase, those patterns become reliable and efficient. The paper characterizes this as a general pattern rather than an artifact of their specific hyperparameter choices, which matters if it generalizes: it would mean that planning zero RL runs requires budgeting for both phases rather than assuming quality improves monotonically with compute.
The emergent behaviors are the part of the paper worth reading carefully. Without explicit programming for any of these, Ring-2.5-1T-Zero develops structured formatting, self-verification (checking its own answers before committing), and parallel reasoning (exploring multiple solution paths simultaneously in its chain-of-thought). These make sense as instrumental behaviors: a model optimizing for correct answers learns that checking answers and exploring alternatives are useful. What’s less obvious is “context anxiety” — a term the authors use but don’t precisely define in the abstract. In the context of reasoning models, it likely describes behavior changes when the model operates near its context limit: perhaps becoming repetitive, over-summarizing, or less reliable in reasoning traces when the full context is dense. If that’s the right reading, it suggests the model develops something like an implicit model of its own limitations — awareness that dense context is a harder operating condition — without being told to.
The evaluation uses seven mathematical benchmarks plus a new framework for assessing reasoning trace quality across three dimensions: comprehensibility (can a human follow the chain-of-thought), reproducibility (does the same reasoning pattern reliably produce the same result), and efficiency (does the model reach correct answers without excessive token use). That last dimension is worth noting as a choice. Longer reasoning chains that get correct answers at the cost of many more tokens are a real deployment concern, and most benchmarks don’t penalize them. This one does.
Whether zero RL at 1T parameters generalizes beyond math to other verifiable-reward domains is an open question. Mathematical reasoning has the cleanest programmatic check. Code execution is a reasonable second. The behaviors that emerge in well-defined settings — self-verification, parallel exploration, context awareness — are the same behaviors you’d want in agents operating in less structured domains. Whether they transfer when the reward signal is noisier, or when “correct” is harder to define, is where this line of research goes next.