Grpo · AI Beat

17 Sep 2026 · AI Beat Desk

Teaching a 4B Model to Outplan Postgres

Rohan Bansal trained a 4B parameter model to generate Postgres join-order hints via distillation from a frontier model followed by reinforcement learning with a custom anchored GRPO variant — yielding a 1.81× geometric mean speedup on 113 join-heavy queries for about $1,200 in total compute. The methodology shows how cheap specialist RL is getting for niche performance optimization tasks.

16 Sep 2026 · AI Beat Desk

RL Training Has a Hard Problem Problem

A new paper and blog post from Michael Noukhovitch identifies the "Matthew Effect" in RL training for LLMs: standard GRPO concentrates gains on problems a model can already mostly solve, while genuinely hard problems see minimal improvement. The fix — Never Give Up (NGU) — is an adaptive sampling strategy that keeps generating completions until it finds one that works, naturally redistributing compute toward the problems that need it.

28 Jul 2026 · AI Beat Desk

Five Hundred Dollars of RL

Fermi Sense and Ramp fine-tuned a 9B open-source model with GRPO for $500 and outperformed every frontier configuration on a catalog review task — 87.3% vs 76.9%, at 40x lower inference cost. The benchmark has a Goodhart's Law concern, but the underlying economics of task-specific RL fine-tuning are real and worth taking seriously.