Agentic AI news, research notes, and industry analysis.
Tag
4 stories in this tag.
A new paper finds that reinforcement learning gains in transformers concentrate almost entirely in a narrow band of middle layers. Training just one layer at roughly 40–60% network depth can match or exceed full-parameter RL fine-tuning. The finding challenges the assumption that all layers participate equally in post-training, and has practical implications for compute-efficient alignment.
Read more →A paper submitted yesterday finds that reducing MLP width monotonically from early to late transformer layers — using a cosine schedule — consistently improves performance across three scales and four architectures at zero additional cost. Later layers refine the residual stream rather than transform it, so the standard uniform allocation gives too much capacity to the wrong end of the network.
Read more →CODA, a new paper from Tri Dao and colleagues, extends FlashAttention's core insight — keep data on-chip, avoid DRAM round-trips — to all the non-attention operations in a transformer block. Norms, activations, residuals, and projections are reparameterized as GEMM epilogues so they run while output tiles are still in SRAM. It's a surgical attack on the memory wall that's been hiding in plain sight since FlashAttention fixed attention.
Read more →Two papers published on April 24 together give the most precise picture yet of looped transformer architectures — where the same block is reused across depth instead of stacking unique layers. The first derives a recurrence-equivalence exponent φ = 0.46 from 116 training runs, showing that looping carries a real compute cost. The second proposes Hyperloop Transformers, adding hyper-connections to partially recover from it, and demonstrates that a 579M Hyperloop model outperforms a standard 1B transformer on perplexity and downstream benchmarks.
Read more →