Agentic AI news, research notes, and industry analysis.
Tag
64 stories in this tag.
Two tutorial posts on Hacker News today trace the two distinct technical lineages of diffusion language models — discrete masked and continuous embedding-space — both of which have now reached production deployments with 2–10× throughput gains over autoregressive models. The quality gap with AR is measurably narrowing.
Read more →A new paper shows that attention magnitude has near-zero correlation with a token's actual causal contribution to the output — the foundational assumption behind most KV cache eviction methods. TwinKV proposes a training-free "repair pass" using pairwise key redundancy instead, composable with any existing eviction policy.
Read more →Tencent released Hy4 preview under Apache 2.0 on August 28: 770B total parameters, 49B active, 256 routed experts per layer with top-8 routing, 1M context, and a built-in speculative-decoding layer. The architecture makes some interesting bets — and the ratio of total to active compute is the most aggressive we've seen at this scale.
Read more →At Hot Chips 2026 on August 24, Nvidia gave its most detailed technical breakdown of the Vera CPU: 88 Olympus cores on six chiplets, statically partitioned spatial multithreading instead of traditional SMT, a graph prefetcher delivering 3× improvement on graph traversal, and 1.8× on agentic benchmarks over current x86 server CPUs. The design choices reveal what Nvidia thinks "agentic workloads" actually demand from silicon.
Read more →Boyd Kane's essay argues that inference engines — complex parsing stacks supporting hundreds of model architectures — represent an underappreciated attack surface where a malicious model's output could be mistaken for code to execute. CVE-2025-9141 in vLLM, which passed Qwen3 Coder tool-call arguments directly to eval(), shows this isn't theoretical.
Read more →Nari Labs walked through five coordinated optimizations that bring Qwen3-TTS 1.7B to sub-50 ms p95 time-to-first-audio on a single H100, at $2 per million characters — against ElevenLabs at $100/M. None of the five changes require a new model architecture. Each targets a specific latency source, and the gains compound.
Read more →FreeToken, from a team at UC Berkeley and MIT, proposes a serving stack that treats a personal machine's CPU, GPU, and RAM as a single elastic compute surface, adapting to what's actually available rather than committing to a fixed offloading strategy. The result: a 35B model on an 8GB laptop GPU, 284B on a gaming desktop, and the 753B GLM-5.2 on a single workstation with one high-end GPU.
Read more →Cerebras unveiled the CS-4 on August 18 without changing its WSE-3 wafer at all — the gains come from moving power conversion 100× closer to the die and adding a third wafer per rack. The result is a system claiming 4,400 tokens/sec/user, a number that is hard for GPU clusters to match at low batch sizes, along with a 125–135 kW rack TDP that represents the real engineering bet.
Read more →A Vectoral research series maps the gray-market supply chain that routes Western AI model access to Chinese buyers at discounts up to 97.8% off official pricing. The infrastructure is four layers deep, runs on open-source gateways, and poses a model-distillation risk that goes beyond simple revenue loss for providers.
Read more →Google's August 14 post on HEIR describes four production deployments of homomorphic encryption for ML inference — fraud detection, recommendations, network intrusion detection, hotword detection — with hardware accelerator partnerships starting to close the latency gap that has kept HE in the lab for the past decade.
Read more →A preprint from Salesforce and UIUC shows that a strong model can construct inference-time "harnesses" — structural scaffolds applied to a weaker model at test time — that nearly double Theory-of-Mind accuracy from 0.49 to 0.91 without any fine-tuning. The mechanism isn't more reasoning; it's converting unstable reasoning into deterministic code, routing, and format enforcement.
Read more →LFM2.5-2.6B and Needle2 arrived this week at opposite ends of the weight-class spectrum — one trimmed but architecturally orthodox, the other stripped of its feed-forward layers entirely — and together they define the two credible paths to running a real tool-calling agent on constrained hardware.
Read more →Meta's Muse Glimmer is a 30B open-weight model distilled from Muse Spark 1.2, targeting local agent workflows on a single consumer GPU. It slots in above LFM2.5-2.6B and Needle2 in the increasingly crowded on-device agent tier, with Apache 2.0 licensing, hybrid attention, and speculative decoding via a dedicated drafter model.
Read more →Salvatore Sanfilippo — the author of Redis — published a native C+Metal inference engine for MiniMax H3 targeting M3/M5 Macs, roughly repeating what llama.cpp did for language models: bypass the Python stack, write tight hardware-specific kernels, and find out how fast the silicon can actually go.
Read more →OpenChamber v1.18.0, an open-source agentic IDE that lets you run the same task across up to five models and fuse the strongest results, ships a guided diff walkthrough that reorders large changesets into explained stops. It's a privacy-first alternative to commercial tools: code stays local, API keys are yours, and the project now supports any OpenAI-compatible endpoint.
Read more →Databricks talked to engineering leaders at Stripe, Coinbase, Uber, and Ramp and wrote up what they're doing about AI coding costs at scale. The playbook looks a lot like cloud cost management circa 2013: smart routing, caching, vendor-neutral abstraction layers, and progressive controls instead of hard caps. 30% cost reduction from routing; ~50% token reduction from compaction. The infrastructure is now real enough to need its own infrastructure.
Read more →AMD's acquisition of Toronto startup Taalas bets on a structural alternative to GPU-based inference: model-specific integrated circuits that etch weights into mask-ROM on the die itself, eliminating the memory-bandwidth wall that constrains all-general-purpose accelerators. Taalas's HC1 claimed 17,000 tok/s for Llama 3.1 8B at one-tenth the power of an H200. The tradeoff is inflexibility — a finished chip runs exactly one model — but AMD sees a disaggregated future where Taalas handles token generation and Instinct GPUs handle prefill.
Read more →DeepGrove's Maple-Preview is a 20B ternary-weight MoE reasoning model trained from scratch at {-1, 0, +1} precision — not quantized down from float. The 5.31 GB checkpoint runs at 218 tokens/second on an M4 Mac mini and 120 tok/s on iPhone, with competitive AIME and GPQA-D scores and an MIT license.
Read more →Swiftlet is a new Swift+Metal runtime that runs 80B-parameter Qwen3-Next in 4.3 GB of RAM on an M5 Mac — not through compression magic, but by exploiting a structural property of MoE models: they activate only about 3B parameters per token regardless of total size. The missing piece is fast enough SSD I/O to stream expert weights on demand, which Apple silicon happens to provide.
Read more →Wafer.ai's benchmarks of Kimi K3 on AMD MI355X tell a story that goes beyond the numbers: the hardware was capable all along, but two ROCm bugs were blocking it. Fixing them yields 48 tok/s per GPU-dollar, against 7 for B200 — a gap that challenges the assumption that NVIDIA owns frontier model inference.
Read more →SQLiteAI released WASTE, a dependency-free C inference engine that runs Kimi K3's 2.78-trillion-parameter model on 29GB of RAM at 0.50 tok/s by keeping the resident trunk in memory and streaming activated experts from NVMe with a single pread() per expert.
Read more →TurboFieldfare, a Swift/Metal inference engine posted to Hacker News overnight, runs Gemma 4 26B in roughly 2 GB of RAM by keeping the model's shared core in memory and streaming routed experts from SSD via explicit pread I/O — achieving 31–35 tok/s on an M5 MacBook Pro, well into interactive usability. It's the expert-streaming technique from Colibri, applied to a smaller MoE on purpose-built Apple hardware, and the throughput gap shows what platform-specific implementation buys.
Read more →TracerML's Echo, in public alpha today, routes each request across a coordinated fleet of open-weight models and claims Fable-comparable results at one-third the inference cost. The core insight isn't just "use cheaper models when you can" — it's that models are complementary, and a weaker model overall can still outperform a stronger one on specific problem types.
Read more →Marcel Røed released GigaToken, a Rust tokenizer using SIMD and cache-optimized byte-pair encoding that runs 500–1,000× faster than HuggingFace's tokenizers and up to 681× faster than tiktoken — reaching 24 GB/s on a single CPU.
Read more →PrismML released Bonsai 27B on July 14: 1-bit binary and ternary builds of Qwen3.6-27B that fit in 3.9 GB and 5.9 GB respectively, run at 11 tok/s on an iPhone 17 Pro, and retain over 90% and 95% of full-precision benchmark performance. The compression factor is around 14× versus FP16, and the models are available under Apache 2.0.
Read more →Mesh LLM, published yesterday on the iroh blog, routes LLM inference across a peer-to-peer mesh with no central coordinator — requests go locally, to a peer that already has the model loaded, or split by layer range across multiple nodes via the "Skippy" engine. It works well on a LAN and becomes impractical across the internet, for a predictable reason.
Read more →Colibri, a ~1300-line pure-C engine posted on Hacker News overnight, runs the 744B GLM-5.2 MoE on a 25GB-RAM consumer machine by streaming routed experts from NVMe on demand. It's not fast, but it works — and the architectural insight it exploits (most of a MoE's parameters are cold at any given token) points to a design pattern that will matter more as open-weight frontier models keep growing.
Read more →Meituan's LongCat-2.0 — a 1.6T-parameter open-weight MoE trained entirely on domestic Chinese ASICs — spent two months deployed anonymously on OpenRouter as "Owl Alpha," quietly reaching #1 on Hermes Agent and #2 on Claude Code before the company claimed it. The reveal is technically notable, but the verification gaps are worth keeping in view.
Read more →DeepSeek released DSpark on June 27 — a semi-parallel speculative decoding framework already running in production for DeepSeek-V4 — alongside DeepSpec, an MIT-licensed toolkit packaging three drafting algorithms with complete training and evaluation pipelines. Together they let anyone train a custom draft model for their own target LLM, not just the models DeepSeek ships.
Read more →Qualcomm agreed to acquire Modular for approximately $3.9 billion on June 24. Modular makes Mojo (a Python-superset systems language) and MAX (a hardware-agnostic inference engine). The deal is a bet that AI inference will fracture across hardware vendors, and whoever owns the abstraction layer wins.
Read more →FUTO released the models behind their swipe keyboard — a three-component stack totalling 2.5 million parameters that achieves 26% fewer errors than Gboard on their benchmark. It trains on one workstation GPU, runs on low-end Android devices in milliseconds, and is the first freely licensed open swipe-typing model. It's a reminder that model scale is a tool, not an objective.
Read more →Google's DiffusionGemma 26B-A4B is a discrete text diffusion model that generates tokens in parallel blocks rather than left-to-right, hitting 1100+ tokens/sec on a single H100 and fitting in 18 GB of VRAM quantized. It's open under Apache 2.0 and marks the first time a production-quality diffusion LM from a major lab lands on consumer hardware — with real benchmark results showing what you trade away for that speed.
Read more →OpenCV 5.0 ships a ground-up rewrite of its DNN engine: ONNX operator coverage jumps from 22% to 80%+, and native LLM/VLM support lands in a library already deployed across embedded systems, medical devices, and industrial hardware that can't run PyTorch.
Read more →Xiaomi and TileRT published MiMo-V2.5-Pro-UltraSpeed on June 8, pushing a one-trillion-parameter model past 1000 tokens per second on a single standard 8-GPU node — no custom silicon, just three carefully chosen co-design decisions applied to a commodity cluster.
Read more →NVIDIA's RTX Spark puts a Blackwell GPU and full CUDA stack inside a laptop SoC — enough to run a 120B-parameter model locally with 1M-token context. At roughly the same moment, Perplexity shipped a hybrid inference orchestrator that uses a compact on-device model to automatically decide which tasks stay local and which escalate to the cloud. Together they sketch what a local-AI platform actually looks like in hardware and software.
Read more →Two papers published this week attack the KV cache memory bottleneck from opposite directions: one proposes sharing key and value projections at training time for a 50% cache reduction with 3.1% perplexity cost, the other quantizes stored cache values to 4-bit keys and 2-bit values with no calibration required and throughput above FP16. Together they suggest the cache is far more compressible than inference engineers typically assume.
Read more →Google's Magenta RealTime 2 cuts live music generation control latency from ~3 seconds to ~200ms by shifting from chunk-based to frame-level causal processing. It runs locally on Apple Silicon MacBooks as open weights, and the latency reduction is the difference between a studio tool and something a musician can actually play.
Read more →A detailed engineering account of bringing DeepSeek-V4-Flash up on AMD MI300X reveals the real cost of AMD's software ecosystem gaps: FP8 format fragmentation, missing kernels, and HIP graph constraints that each required dedicated engineering effort before getting to 2,700 tokens/s.
Read more →PrismML's Bonsai Image 4B applies 1-bit and ternary quantization to a FLUX.2 Klein diffusion transformer, compressing it 8.3× to 0.93 GB — small enough to generate images on an iPhone in under 10 seconds. It's the first demonstration that extreme quantization techniques developed for language models transfer cleanly to diffusion architectures.
Read more →Liquid AI ships LFM2.5-8B-A1B, a 38T-token trained hybrid model where 18 of 24 layers are gated convolution blocks rather than attention — and it reaches 253 tokens/second on an M5 Max CPU with under 6 GB of memory.
Read more →Token prices are falling fast, but enterprise AI bills are rising. Uber burned through its entire 2026 AI coding budget in four months driven by Claude Code adoption. Goldman Sachs projects a 24× increase in token consumption by 2030. The Jevons paradox shows up again: efficiency gains don't reduce consumption — they expand it.
Read more →CODA, a new paper from Tri Dao and colleagues, extends FlashAttention's core insight — keep data on-chip, avoid DRAM round-trips — to all the non-attention operations in a transformer block. Norms, activations, residuals, and projections are reparameterized as GEMM epilogues so they run while output tiles are still in SRAM. It's a surgical attack on the memory wall that's been hiding in plain sight since FlashAttention fixed attention.
Read more →δ-mem augments a frozen full-attention LLM with an 8×8 associative memory state updated by delta-rule learning, applying low-rank corrections to attention at inference time — no fine-tuning required. It reaches 1.31× gains on memory-heavy benchmarks and 1.20× on long-conversation tasks.
Read more →NVIDIA's SANA-WM generates 60-second, 720p video from a single image and a camera trajectory — on a single GPU. The open-source 2.6B-parameter model achieves 36× higher throughput than prior open-source world models and ships under Apache 2.0.
Read more →Mistletoe (arXiv 2605.14005) demonstrates a stealthy adversarial attack on speculative decoding systems: craft inputs that look normal to the target model but cause the draft model to disagree, collapsing acceptance length and throughput while leaving output quality and perplexity unchanged. The attack exploits the fundamental gap between draft and target distributions that all speculative systems rely on bridging.
Read more →Orthrus (arXiv 2605.12825) grafts a trainable diffusion head onto a frozen AR backbone, sharing the exact same KV cache. An intra-model consensus mechanism guarantees that every accepted token matches the AR distribution exactly — no approximation, no quality tradeoff — while achieving up to 7.8× speedup on Qwen3-8B with only O(1) memory overhead. The approach sidesteps the core operational cost of speculative decoding: maintaining a separate, carefully calibrated draft model.
Read more →Cactus Compute released Needle, a 26M-parameter MIT-licensed model for on-device function calling that strips out all feed-forward networks from the transformer. The architectural choice is a thesis: tool calling is retrieval-and-routing, not reasoning, and attention is the right primitive for it. The numbers are striking — 6000 tok/s prefill on consumer hardware — even if the playground has rough edges.
Read more →NVIDIA's NVlabs released cuda-oxide v0.1.0 on May 7, an experimental compiler that takes standard Rust and emits NVIDIA PTX directly — no CUDA C++, no DSLs, no foreign language bindings. The pipeline goes through a custom rustc codegen backend and a Rust-native MLIR-like IR called Pliron. Alpha-stage and Linux-only, but it signals where NVIDIA thinks GPU kernel development might eventually land.
Read more →A University of Washington paper shows a multi-agent loop that generates complete LLM serving systems end-to-end. On standard workloads it matches vLLM; on six specialized scenarios — hybrid architectures, streaming ASR, constrained decoding, multimodal pipelines — it beats it by 1.7× to nearly 6×. The paper surfaces a practical claim: the general-purpose serving stack is a compromise, and specialization can be automated.
Read more →Salvatore Sanfilippo (antirez, Redis) released ds4: a single-model Metal inference engine for DeepSeek V4 Flash that deliberately rejects the general-framework approach. Asymmetric 2-bit quantization on MoE experts only gets a 280B-parameter model into 128 GB RAM with 26–36 t/s generation, 1M-token context, and disk-persisted KV cache on Apple Silicon.
Read more →Google ships multi-token prediction draft models for the full Gemma 4 family under Apache 2.0, reporting up to 3x throughput gains. The architecture is tightly coupled — shared embeddings, last-layer activations — which keeps the drafter accurate but limits reuse. MoE variants complicate the picture.
Read more →Moonshot AI ships Kimi K2.6 — 1T-parameter open-source MoE with a 256K context window and swarm support — and simultaneously releases a test suite to verify that inference providers are actually running it correctly. The same day, Alibaba closes off Qwen3.6-Max. Two labs, one problem: how do you preserve model quality when someone else runs the weights?
Read more →A blog post published April 18 describes a technique for running LLM inference inside a WebAssembly sandbox at near-native GPU speed on Apple Silicon. By overriding Wasmtime's memory allocator to back Wasm linear memory with a Metal buffer via makeBuffer(bytesNoCopy:), the author collapses the Wasm–GPU boundary entirely: 0.03 MB overhead vs 16.78 MB for the copy approach, ~9 ms/token for Llama 3.2 1B on M1, and KV cache snapshots that restore 5.45× faster than recomputing prefill.
Read more →Qwen3.6-35B-A3B landed on April 16 under Apache 2.0 — 35 billion total parameters, 3 billion active per token, and a hybrid architecture that alternates Gated DeltaNet linear attention with standard attention blocks. It runs on a laptop, scores 73.4 on SWE-bench Verified, and the architecture is more interesting than the benchmark numbers alone suggest.
Read more →Eigen Labs — the team behind EigenLayer Ethereum restaking — launched Darkbloom on April 15: a research-preview decentralized inference network that routes AI requests through idle Apple Silicon Macs with cryptographic privacy guarantees. The node operator genuinely cannot read your prompt. The security model is layered and interesting; the economics are aggressive; the project is very early.
Read more →A new paper from a mix of academic and industry researchers identifies why diffusion language models consistently trail their autoregressive counterparts despite strong theoretical properties: they don't agree with what they generate. The proposed fix — Introspective Strided Decoding — lets an 8B DLM match same-scale AR quality while running 2.9–4.1x faster at high concurrency.
Read more →A new preprint identifies a consistent pattern in large reasoning models: the first generated solution outperforms later alternatives, and continued reasoning can actively degrade accuracy. The proposed fix, called RED, improves performance by up to 19% while cutting token usage by 37–70% versus competitive baselines. It's a useful challenge to the assumption that more inference compute is always better.
Read more →Microsoft released three foundational AI models through Azure AI Foundry on April 2: MAI-Transcribe-1 for speech, MAI-Voice-1 for synthesis, and MAI-Image-2 for generation. These are Microsoft's first internally built foundational models — a quiet but significant signal that the company wants more control over its AI stack than the OpenAI partnership alone provides.
Read more →MLPerf Inference v6.0 results show NVIDIA achieved a 2.77x throughput improvement on DeepSeek-R1 since the v5.1 results six months ago — on the same B200 hardware. The gains came entirely from software: disaggregated prefill/decode serving, kernel fusion, pipelined execution, and multi-token prediction. Token cost dropped to $0.30/M. It's a useful reminder that the current inference scaling curve has two axes, and software is doing more work than it gets credit for.
Read more →Arcee AI released Trinity Large Thinking on April 1 — the reasoning-optimized variant of their 400B sparse MoE, trained by a 30-person startup on 2,048 Nvidia B300 GPUs. It ranks #2 on PinchBench for agentic tasks at roughly 96% lower cost than the top model, under Apache 2.0. The architecture — 256 experts with 4 active per token — is worth understanding.
Read more →Ollama's preview MLX backend replaces direct Metal calls on Apple Silicon with Apple's dedicated ML framework, yielding a 93% decode speedup for Qwen3.5-35B-A3B on M5 chips. The update also adds NVFP4 quantization and a smarter KV cache — including prefix-aware eviction that keeps shared system prompts hot across conversations.
Read more →CERN has been running AI models on FPGAs at the LHC for years, but a Register piece this week described the system in detail. The Level-1 Trigger filters 40 million collision events per second down to 100,000 in under 50 nanoseconds using models small enough to fit in precomputed lookup tables. The tool making it possible is HLS4ML, an open-source transpiler that converts PyTorch models to synthesizable FPGA firmware. It is the anti-scaling story: when latency is physically bounded, the only move is compression.
Read more →Arm's first production AI CPU, Google's TurboQuant, and Hypura's NVMe-first runtime converge on memory bandwidth as the core inference bottleneck.
Read more →Flash-MoE shows how SSD-streamed experts let a 397B-parameter MoE run locally on consumer Apple Silicon hardware.
Read more →