Ray Closes the Loop on KV-Aware Routing
Ray 2.58.0, released August 23, completes the KV-cache-aware request routing work previewed in 2.57: tokenization now happens on the LLMRouter ingress replica, tokens are passed out-of-band to the engine, KV lifecycle events broadcast to all routers, and CPU-offloaded cache blocks count toward cache hit scoring.
Read more →