Agentic AI news, research notes, and industry analysis.
Tag
4 stories in this tag.
Desert Ant Labs launched 18 small AI models for audio, vision, and text that run entirely on-device — no cloud API, no per-token cost, free up to 100,000 monthly active devices. Their Voz transcription model hits 4.7x Whisper speed on an iPhone; a language identifier does its job in 2MB. The economics of AI features in apps look different when inference is a fixed sunk cost.
Read more →Deltafin, a Rust binary for consumer inference, now runs the full unmodified Kimi K3 2.8-trillion-parameter MoE model on an Apple Silicon MacBook at around 1 token per second, streaming 1.45TB of expert weights from four external SSDs. It took six weeks to go from 0.014 tok/s to 1.0.
Read more →FreeToken, from a team at UC Berkeley and MIT, proposes a serving stack that treats a personal machine's CPU, GPU, and RAM as a single elastic compute surface, adapting to what's actually available rather than committing to a fixed offloading strategy. The result: a 35B model on an 8GB laptop GPU, 284B on a gaming desktop, and the 753B GLM-5.2 on a single workstation with one high-end GPU.
Read more →A blog post published April 18 describes a technique for running LLM inference inside a WebAssembly sandbox at near-native GPU speed on Apple Silicon. By overriding Wasmtime's memory allocator to back Wasm linear memory with a Metal buffer via makeBuffer(bytesNoCopy:), the author collapses the Wasm–GPU boundary entirely: 0.03 MB overhead vs 16.78 MB for the copy approach, ~9 ms/token for Llama 3.2 1B on M1, and KV cache snapshots that restore 5.45× faster than recomputing prefill.
Read more →