Attention · AI Beat

13 Jul 2026 · AI Beat Desk

Open Kernels for Sparse Attention Training

Flash-MSA, published July 11, provides the first open-source performant training kernels for MiniMax Sparse Attention — the block-sparse attention mechanism that enabled M3's 28.4× compute reduction at 1M context. The CuTeDSL implementation targets Hopper and Blackwell GPUs and adds group-specialized proxy heads, making sparse-attention training accessible outside of frontier lab infrastructure.

02 Jun 2026 · AI Beat Desk

MiniMax M3 and the Cost of Long Context

MiniMax M3 launches with a sparse attention mechanism that cuts per-token compute at 1M tokens to one-twentieth of its predecessor. The architecture is genuinely interesting; the benchmarks require scrutiny; the license is almost certainly not what the word "open-weight" implies.

17 May 2026 · AI Beat Desk

Sixty-Four Cells of Memory

δ-mem augments a frozen full-attention LLM with an 8×8 associative memory state updated by delta-rule learning, applying low-rank corrections to attention at inference time — no fine-tuning required. It reaches 1.31× gains on memory-heavy benchmarks and 1.20× on long-conversation tasks.