The original Magenta RealTime had an inconvenient property: control latency around three seconds. That is fine for studio exploration where you are directing — set a prompt, wait, listen, adjust — but it is incompatible with live musical interaction. Three seconds after you change a parameter, something different happens. That is not an instrument; it is a slow feedback loop.
Magenta RealTime 2, released June 4 by Google, reduces that to approximately 200ms. The difference is architectural. The original model processed audio in 2-second chunks, meaning any change in the control signal had to wait for the current chunk to complete before the next generation step could incorporate it. MRT2 shifts to frame-level processing: audio is generated in 40ms frames, and conditioning (text, audio, MIDI) is applied at every frame boundary. Control latency is now determined by frame size plus inference overhead rather than chunk size.
The model behind this is decoder-only with causal sliding window attention, replacing the T5-style bidirectional encoder from the original. Bidirectional encoders see the full input sequence in both directions before producing output — useful for understanding tasks, awkward for streaming generation where you cannot wait for future frames. A causal decoder with a sliding window processes each incoming frame without future context, which is what makes the latency target achievable. The audio codec is SpectroStream, compressing 48kHz stereo to 3kbps at a 25Hz frame rate — the model operates on token-sized representations rather than raw waveforms, keeping the compute tractable.
The 2.4B base model requires a MacBook M3 Pro or M2 Max at minimum for real-time generation. The 230M small model runs on any Apple Silicon MacBook. The inference engine is a C++ library with MLX backing, distributed via pip install magenta-rt. Weights are on Hugging Face.
Control is multi-modal: MIDI input, text prompts, and audio conditioning work simultaneously. Drum on/off is a named control, which matters in live settings — toggling a drum pattern without retyping a full text prompt is the kind of practical affordance that distinguishes a tool built with actual use in mind. Style blending and sound cloning from a reference audio clip are also supported.
The quality of the generated audio in the demos is competent rather than exceptional — it sounds like capable AI music generation, not like something that would fool a listener with no prior knowledge. But the more important benchmark here is latency and portability. Generative audio models at this capability level have generally required cloud GPU or TPU access. Running locally on a laptop you might actually take to a rehearsal is a different category of deployment.
The limitations are real. It is Mac-only, and the base model requirement of M3 Pro or M2 Max excludes a lot of Apple Silicon hardware. There is no mention of Windows or Linux support. The C++ inference engine is open source, so a port is possible, but it is not on any published timeline. And 200ms, while playable, is still noticeable — classically-trained musicians can distinguish latencies above around 10ms in direct playing contexts. For generative accompaniment and ambient steering it is fine; for tight rhythmic call-and-response it is the ceiling of what works.
For musicians who own an M3 Pro MacBook and are curious about what live AI accompaniment actually feels like in practice rather than in demo videos, this is worth an afternoon. Everything needed — weights, library, example instruments — is on GitHub.