Kernels · AI Beat

13 Jul 2026 · AI Beat Desk

Open Kernels for Sparse Attention Training

Flash-MSA, published July 11, provides the first open-source performant training kernels for MiniMax Sparse Attention — the block-sparse attention mechanism that enabled M3's 28.4× compute reduction at 1M context. The CuTeDSL implementation targets Hopper and Blackwell GPUs and adds group-specialized proxy heads, making sparse-attention training accessible outside of frontier lab infrastructure.

22 May 2026 · AI Beat Desk

The Rest of the Transformer, Fused

CODA, a new paper from Tri Dao and colleagues, extends FlashAttention's core insight — keep data on-chip, avoid DRAM round-trips — to all the non-attention operations in a transformer block. Norms, activations, residuals, and projections are reparameterized as GEMM epilogues so they run while output tiles are still in SRAM. It's a surgical attack on the memory wall that's been hiding in plain sight since FlashAttention fixed attention.