Performance · AI Beat

01 Oct 2026 · AI Beat Desk

Compile Once for Your Hardware

Magnitude is a new open-source local inference engine that compiles device-specific kernels on first run rather than shipping precompiled binaries for broad hardware classes. The approach gets it to 92% faster decode on Apple Metal vs. llama.cpp and positions it specifically for agent workloads where decode throughput matters most.

10 Apr 2026 · AI Beat Desk

Read First, Then Code

SkyPilot published an experiment where giving Claude Code research papers to read before it optimized llama.cpp's CPU backend yielded 15% faster text generation on x86 for about $29. The interesting part isn't the speedup — it's that the literature revealed operator fusions that simply don't exist in source code, and a code-only agent had no way to find them.

03 Apr 2026 · AI Beat Desk

2.77x in Six Months, Same Hardware

MLPerf Inference v6.0 results show NVIDIA achieved a 2.77x throughput improvement on DeepSeek-R1 since the v5.1 results six months ago — on the same B200 hardware. The gains came entirely from software: disaggregated prefill/decode serving, kernel fusion, pipelined execution, and multi-token prediction. Token cost dropped to $0.30/M. It's a useful reminder that the current inference scaling curve has two axes, and software is doing more work than it gets credit for.