TurboFieldfare, a Swift/Metal inference engine posted to Hacker News overnight, runs Gemma 4 26B in roughly 2 GB of RAM by keeping the model's shared core in memory and streaming routed experts from SSD via explicit pread I/O — achieving 31–35 tok/s on an M5 MacBook Pro, well into interactive usability. It's the expert-streaming technique from Colibri, applied to a smaller MoE on purpose-built Apple hardware, and the throughput gap shows what platform-specific implementation buys.
Cactus Hybrid adds a confidence probe to Gemma 4 that reads internal activations to score each completion 0–1 and routes low-confidence queries to a cloud model. 65–85% of queries stay on-device; overall accuracy matches Gemini 3.1 Flash-Lite. The probe generalizes to audio (0.79–0.88 AUROC) despite no audio training data.
Google released quantization-aware training checkpoints for Gemma 4 with a new mobile-specific format — channel-wise quantization aligned with NPU memory layouts, 2-bit compression for token generation layers, pre-calculated scaling constants — bringing the Gemma 4 E2B text model under 1 GB of memory.
Google ships multi-token prediction draft models for the full Gemma 4 family under Apache 2.0, reporting up to 3x throughput gains. The architecture is tightly coupled — shared embeddings, last-layer activations — which keeps the drafter accurate but limits reuse. MoE variants complicate the picture.
