Gpu · AI Beat

22 May 2026 · AI Beat Desk

The Rest of the Transformer, Fused

CODA, a new paper from Tri Dao and colleagues, extends FlashAttention's core insight — keep data on-chip, avoid DRAM round-trips — to all the non-attention operations in a transformer block. Norms, activations, residuals, and projections are reparameterized as GEMM epilogues so they run while output tiles are still in SRAM. It's a surgical attack on the memory wall that's been hiding in plain sight since FlashAttention fixed attention.

12 May 2026 · AI Beat Desk

NVIDIA's cuda-oxide Wants GPU Kernels Written in Rust

NVIDIA's NVlabs released cuda-oxide v0.1.0 on May 7, an experimental compiler that takes standard Rust and emits NVIDIA PTX directly — no CUDA C++, no DSLs, no foreign language bindings. The pipeline goes through a custom rustc codegen backend and a Rust-native MLIR-like IR called Pliron. Alpha-stage and Linux-only, but it signals where NVIDIA thinks GPU kernel development might eventually land.