Pretraining · AI Beat

23 Jun 2026 · AI Beat Desk

Give Early Layers More

A paper submitted yesterday finds that reducing MLP width monotonically from early to late transformer layers — using a cosine schedule — consistently improves performance across three scales and four architectures at zero additional cost. Later layers refine the residual stream rather than transform it, so the standard uniform allocation gives too much capacity to the wrong end of the network.

24 Apr 2026 · AI Beat Desk

Generation Is Pretraining, in Vision Too

Google DeepMind's Vision Banana paper shows that training a model to generate images — and only that — produces transferable visual representations strong enough to beat specialized discriminative models on segmentation and metric depth estimation when lightly instruction-tuned. The finding is the visual analog of how LLM pretraining generalizes across language tasks.