Train on Your Best Guess
A new pretraining technique called Explorative Modeling adds a best-of-K selection step to the training loop — generate K candidates, keep the one closest to the target, backprop through only that one. The efficiency gains on image and video models are large and grow with scale, suggesting a genuine third axis alongside parameters and data. For autoregressive LLMs the gains are modest for now, but the underlying idea is worth watching.
Read more →