Multimodal · AI Beat

07 Jun 2026 · AI Beat Desk

When One Model Reasons and Simulates

NVIDIA's Cosmos 3 bets on collapsing the physical AI model stack — VLM understanding, video world simulation, and robot action generation — into a single Mixture-of-Transformers architecture where reasoning and diffusion paths share joint attention. The key question is whether that coupling actually beats specialist models, or whether this is mainly a convenience story.

04 Jun 2026 · AI Beat Desk

Gemma 4 12B Goes Encoder-Free

Google DeepMind's Gemma 4 12B discards the conventional encoder-stack approach to multimodal models, feeding raw pixel patches and audio waveforms directly into the LLM backbone through lightweight linear projections. The result fits in 16 GB of RAM, accepts native audio, and fine-tunes as a single unified model.

14 May 2026 · AI Beat Desk

Dropping the Encoder

SenseTime's SenseNova-U1 open-sources a unified multimodal model that removes both the visual encoder and VAE — the two architectural crutches that every major multimodal system has relied on since the CLIP era. The NEO-unify architecture processes pixels natively through a shared transformer backbone, with a direct pixel-space MLP head for generation. Benchmarks on image generation and interleaved content put it at or above current open-source leaders, with the spatial reasoning numbers being the most credible differentiator.

03 May 2026 · AI Beat Desk

Drop the Encoder: Meta's Tuna-2 Goes Straight to Pixels

Meta AI's Tuna-2 paper shows that a 7B unified multimodal model trained end-to-end on raw pixel patches — with no pretrained vision encoder — matches or beats its CLIP-based sibling at scale, particularly on fine-grained perception tasks. The result challenges a design assumption that has been stable in multimodal modeling for years.

24 Apr 2026 · AI Beat Desk

Generation Is Pretraining, in Vision Too

Google DeepMind's Vision Banana paper shows that training a model to generate images — and only that — produces transferable visual representations strong enough to beat specialized discriminative models on segmentation and metric depth estimation when lightly instruction-tuned. The finding is the visual analog of how LLM pretraining generalizes across language tasks.