Agentic AI news, research notes, and industry analysis.
Category
22 stories in this category.
Nari Labs walked through five coordinated optimizations that bring Qwen3-TTS 1.7B to sub-50 ms p95 time-to-first-audio on a single H100, at $2 per million characters — against ElevenLabs at $100/M. None of the five changes require a new model architecture. Each targets a specific latency source, and the gains compound.
Read more →FreeToken, from a team at UC Berkeley and MIT, proposes a serving stack that treats a personal machine's CPU, GPU, and RAM as a single elastic compute surface, adapting to what's actually available rather than committing to a fixed offloading strategy. The result: a 35B model on an 8GB laptop GPU, 284B on a gaming desktop, and the 753B GLM-5.2 on a single workstation with one high-end GPU.
Read more →Cerebras unveiled the CS-4 on August 18 without changing its WSE-3 wafer at all — the gains come from moving power conversion 100× closer to the die and adding a third wafer per rack. The result is a system claiming 4,400 tokens/sec/user, a number that is hard for GPU clusters to match at low batch sizes, along with a 125–135 kW rack TDP that represents the real engineering bet.
Read more →Stripe has agreed to acquire OpenRouter for over $7 billion — a company whose CEO described it as "the equivalent of Stripe for AI." The deal puts a single payment processor in control of the routing layer that sits between developers and 400+ AI models, raising real questions about what a non-neutral router means for the model access market.
Read more →LFM2.5-2.6B and Needle2 arrived this week at opposite ends of the weight-class spectrum — one trimmed but architecturally orthodox, the other stripped of its feed-forward layers entirely — and together they define the two credible paths to running a real tool-calling agent on constrained hardware.
Read more →Salvatore Sanfilippo — the author of Redis — published a native C+Metal inference engine for MiniMax H3 targeting M3/M5 Macs, roughly repeating what llama.cpp did for language models: bypass the Python stack, write tight hardware-specific kernels, and find out how fast the silicon can actually go.
Read more →Databricks talked to engineering leaders at Stripe, Coinbase, Uber, and Ramp and wrote up what they're doing about AI coding costs at scale. The playbook looks a lot like cloud cost management circa 2013: smart routing, caching, vendor-neutral abstraction layers, and progressive controls instead of hard caps. 30% cost reduction from routing; ~50% token reduction from compaction. The infrastructure is now real enough to need its own infrastructure.
Read more →AMD's acquisition of Toronto startup Taalas bets on a structural alternative to GPU-based inference: model-specific integrated circuits that etch weights into mask-ROM on the die itself, eliminating the memory-bandwidth wall that constrains all-general-purpose accelerators. Taalas's HC1 claimed 17,000 tok/s for Llama 3.1 8B at one-tenth the power of an H200. The tradeoff is inflexibility — a finished chip runs exactly one model — but AMD sees a disaggregated future where Taalas handles token generation and Instinct GPUs handle prefill.
Read more →DeepGrove's Maple-Preview is a 20B ternary-weight MoE reasoning model trained from scratch at {-1, 0, +1} precision — not quantized down from float. The 5.31 GB checkpoint runs at 218 tokens/second on an M4 Mac mini and 120 tok/s on iPhone, with competitive AIME and GPQA-D scores and an MIT license.
Read more →Swiftlet is a new Swift+Metal runtime that runs 80B-parameter Qwen3-Next in 4.3 GB of RAM on an M5 Mac — not through compression magic, but by exploiting a structural property of MoE models: they activate only about 3B parameters per token regardless of total size. The missing piece is fast enough SSD I/O to stream expert weights on demand, which Apple silicon happens to provide.
Read more →Wafer.ai's benchmarks of Kimi K3 on AMD MI355X tell a story that goes beyond the numbers: the hardware was capable all along, but two ROCm bugs were blocking it. Fixing them yields 48 tok/s per GPU-dollar, against 7 for B200 — a gap that challenges the assumption that NVIDIA owns frontier model inference.
Read more →SQLiteAI released WASTE, a dependency-free C inference engine that runs Kimi K3's 2.78-trillion-parameter model on 29GB of RAM at 0.50 tok/s by keeping the resident trunk in memory and streaming activated experts from NVMe with a single pread() per expert.
Read more →TurboFieldfare, a Swift/Metal inference engine posted to Hacker News overnight, runs Gemma 4 26B in roughly 2 GB of RAM by keeping the model's shared core in memory and streaming routed experts from SSD via explicit pread I/O — achieving 31–35 tok/s on an M5 MacBook Pro, well into interactive usability. It's the expert-streaming technique from Colibri, applied to a smaller MoE on purpose-built Apple hardware, and the throughput gap shows what platform-specific implementation buys.
Read more →Cactus Hybrid adds a confidence probe to Gemma 4 that reads internal activations to score each completion 0–1 and routes low-confidence queries to a cloud model. 65–85% of queries stay on-device; overall accuracy matches Gemini 3.1 Flash-Lite. The probe generalizes to audio (0.79–0.88 AUROC) despite no audio training data.
Read more →PrismML released Bonsai 27B on July 14: 1-bit binary and ternary builds of Qwen3.6-27B that fit in 3.9 GB and 5.9 GB respectively, run at 11 tok/s on an iPhone 17 Pro, and retain over 90% and 95% of full-precision benchmark performance. The compression factor is around 14× versus FP16, and the models are available under Apache 2.0.
Read more →Inscribe's benchmark of Apple's new SpeechAnalyzer API on macOS 26.5.1 finds it achieves 2.12% word error rate versus Whisper Small's 3.74%, while running three times faster — at the cost of covering roughly 30 languages instead of 100+.
Read more →Mesh LLM, published yesterday on the iroh blog, routes LLM inference across a peer-to-peer mesh with no central coordinator — requests go locally, to a peer that already has the model loaded, or split by layer range across multiple nodes via the "Skippy" engine. It works well on a LAN and becomes impractical across the internet, for a predictable reason.
Read more →Colibri, a ~1300-line pure-C engine posted on Hacker News overnight, runs the 744B GLM-5.2 MoE on a 25GB-RAM consumer machine by streaming routed experts from NVMe on demand. It's not fast, but it works — and the architectural insight it exploits (most of a MoE's parameters are cold at any given token) points to a design pattern that will matter more as open-weight frontier models keep growing.
Read more →Ternlight ships a sentence embedding model as a 7MB WASM bundle that runs on CPU in the browser — no API, no model download, no GPU required. Ternary weights are the key to the footprint; the result is semantic search you can include in an npm install.
Read more →DeepSeek released DSpark on June 27 — a semi-parallel speculative decoding framework already running in production for DeepSeek-V4 — alongside DeepSpec, an MIT-licensed toolkit packaging three drafting algorithms with complete training and evaluation pipelines. Together they let anyone train a custom draft model for their own target LLM, not just the models DeepSeek ships.
Read more →Vicki Boykis published a careful practitioner's report on her local-inference stack this week, and the conclusion that stuck — ~75% of frontier model capability for agentic coding on a 64 GB M2 Mac — is more significant than the raw number suggests. The tooling layer finally grew up, and that changes what "running locally" means.
Read more →CODA, a new paper from Tri Dao and colleagues, extends FlashAttention's core insight — keep data on-chip, avoid DRAM round-trips — to all the non-attention operations in a transformer block. Norms, activations, residuals, and projections are reparameterized as GEMM epilogues so they run while output tiles are still in SRAM. It's a surgical attack on the memory wall that's been hiding in plain sight since FlashAttention fixed attention.
Read more →