Agentic AI news, research notes, and industry analysis.
Tag
6 stories in this tag.
Strata is an MIT-licensed inference engine that runs Qwen3.8-Flash-Next, a 125B-parameter MoE model, on a consumer GPU with 12GB of VRAM at roughly 94 tokens per second. The technique at its core — distributing sparse MoE experts across VRAM, system RAM, and CPU — turns out to be a natural fit for the architecture, not a hack against it.
Read more →Magnitude is a new open-source local inference engine that compiles device-specific kernels on first run rather than shipping precompiled binaries for broad hardware classes. The approach gets it to 92% faster decode on Apple Metal vs. llama.cpp and positions it specifically for agent workloads where decode throughput matters most.
Read more →Prince Canuma, the author of MLX-VLM, shipped v0.0.1 of Nativ: a native SwiftUI app that turns an Apple Silicon Mac into a private, no-subscription local AI server supporting text, vision, audio, and code models via MLX — with OpenAI- and Anthropic-compatible inference endpoints built in.
Read more →NVIDIA's RTX Spark puts a Blackwell GPU and full CUDA stack inside a laptop SoC — enough to run a 120B-parameter model locally with 1M-token context. At roughly the same moment, Perplexity shipped a hybrid inference orchestrator that uses a compact on-device model to automatically decide which tasks stay local and which escalate to the cloud. Together they sketch what a local-AI platform actually looks like in hardware and software.
Read more →Ghost Pepper v2.0.1 is a macOS hold-to-talk tool that quietly chains WhisperKit and a local Qwen 3.5 model to transcribe and clean up speech without any cloud call. It's a small app, but a clear signal of where on-device AI composition is heading.
Read more →Flash-MoE shows how SSD-streamed experts let a 397B-parameter MoE run locally on consumer Apple Silicon hardware.
Read more →