Llm-Serving · AI Beat

24 Aug 2026 · AI Beat Desk

Ray Closes the Loop on KV-Aware Routing

Ray 2.58.0, released August 23, completes the KV-cache-aware request routing work previewed in 2.57: tokenization now happens on the LLMRouter ingress replica, tokens are passed out-of-band to the engine, KV lifecycle events broadcast to all routers, and CPU-offloaded cache blocks count toward cache hit scoring.

16 May 2026 · AI Beat Desk

Speculative Decoding Has an Acceptance Problem You Can Exploit

Mistletoe (arXiv 2605.14005) demonstrates a stealthy adversarial attack on speculative decoding systems: craft inputs that look normal to the target model but cause the draft model to disagree, collapsing acceptance length and throughput while leaving output quality and perplexity unchanged. The attack exploits the fundamental gap between draft and target distributions that all speculative systems rely on bridging.