Agentic AI news, research notes, and industry analysis.
Tag
13 stories in this tag.
Strata is an MIT-licensed inference engine that runs Qwen3.8-Flash-Next, a 125B-parameter MoE model, on a consumer GPU with 12GB of VRAM at roughly 94 tokens per second. The technique at its core — distributing sparse MoE experts across VRAM, system RAM, and CPU — turns out to be a natural fit for the architecture, not a hack against it.
Read more →Low-Zi-Hong sliced a ~0.4B BitNet model across a cluster of seven ESP32S3 microcontrollers connected by daisy-chain SPI: master handles tokenization and embeddings, six compute nodes process four transformer layers each using 1.58-bit ternary weights from flash. It works. It is slow. It is the most literal possible demonstration of the BitNet "run anywhere" claim.
Read more →PrismML's Bonsai 2 27B applies ternary quantization to Qwen3.8 27B and arrives at a 5.9 GB model that retains 98.2% of aggregate benchmark performance — fast enough for interactive use on an RTX GPU or Apple Silicon, Apache 2.0 licensed. At this compression ratio and quality, the case for running full-precision locally mostly disappears.
Read more →Intel researchers surveyed 29 ternary LLM models and found zeros account for up to 51.5% of weights — which means the standard five-trit packing format silently wastes significant storage. BITCOS replaces it with a presence bitmap plus sign vector, reaching 1.485 bits per weight on sparse models and delivering 1.10–1.28× end-to-end decode speedup on Intel CPUs and GPUs.
Read more →Two independent projects landed on Hacker News today pursuing the same idea from different directions: fit something genuinely useful into the smallest possible package. One runs a 28.9M-parameter LLM at 9.5 tok/s on an $8 ESP32-S3 by keeping the embedding table in flash. The other fits complete voice synthesis into 9.36M parameters that run faster than real-time on a laptop CPU. Neither is frontier work. Both are instructive about what constrained-environment AI actually requires.
Read more →PrismML released Bonsai 27B on July 14: 1-bit binary and ternary builds of Qwen3.6-27B that fit in 3.9 GB and 5.9 GB respectively, run at 11 tok/s on an iPhone 17 Pro, and retain over 90% and 95% of full-precision benchmark performance. The compression factor is around 14× versus FP16, and the models are available under Apache 2.0.
Read more →Xiaomi and TileRT published MiMo-V2.5-Pro-UltraSpeed on June 8, pushing a one-trillion-parameter model past 1000 tokens per second on a single standard 8-GPU node — no custom silicon, just three carefully chosen co-design decisions applied to a commodity cluster.
Read more →Google released quantization-aware training checkpoints for Gemma 4 with a new mobile-specific format — channel-wise quantization aligned with NPU memory layouts, 2-bit compression for token generation layers, pre-calculated scaling constants — bringing the Gemma 4 E2B text model under 1 GB of memory.
Read more →Two papers published this week attack the KV cache memory bottleneck from opposite directions: one proposes sharing key and value projections at training time for a 50% cache reduction with 3.1% perplexity cost, the other quantizes stored cache values to 4-bit keys and 2-bit values with no calibration required and throughput above FP16. Together they suggest the cache is far more compressible than inference engineers typically assume.
Read more →PrismML's Bonsai Image 4B applies 1-bit and ternary quantization to a FLUX.2 Klein diffusion transformer, compressing it 8.3× to 0.93 GB — small enough to generate images on an iPhone in under 10 seconds. It's the first demonstration that extreme quantization techniques developed for language models transfer cleanly to diffusion architectures.
Read more →Salvatore Sanfilippo (antirez, Redis) released ds4: a single-model Metal inference engine for DeepSeek V4 Flash that deliberately rejects the general-framework approach. Asymmetric 2-bit quantization on MoE experts only gets a 280B-parameter model into 128 GB RAM with 26–36 t/s generation, 1M-token context, and disk-persisted KV cache on Apple Silicon.
Read more →Two papers published on April 24 together give the most precise picture yet of looped transformer architectures — where the same block is reused across depth instead of stacking unique layers. The first derives a recurrence-equivalence exponent φ = 0.46 from 116 training runs, showing that looping carries a real compute cost. The second proposes Hyperloop Transformers, adding hyper-connections to partially recover from it, and demonstrates that a 579M Hyperloop model outperforms a standard 1B transformer on perplexity and downstream benchmarks.
Read more →PrismML launched Bonsai on March 31, claiming the first commercially viable true 1-bit LLMs: an 8B model that fits in 1.15 GB and runs at 131 tokens/sec on an M4 Pro. The key word is "true" — every layer, including embeddings and attention, is 1-bit, not just the weights in isolation.
Read more →