If you’ve ever tried to run large language model inference at production scale on your own infrastructure, you’ve probably spent time writing glue code that shouldn’t exist. Load-aware routing across GPU pods. Prefix-cache hit rate management. Handling the bursty traffic patterns that coding agents produce — hundreds of concurrent sessions, each with long multi-turn contexts that need to land near their cached state. None of it is hard in theory, and all of it is tedious in practice.
llm-d 1.0, released October 20th at the PyTorch Conference and now a CNCF sandbox project, is an attempt to make this layer standard. IBM, Red Hat, Google, CoreWeave, and NVIDIA founded it in 2025. It cleared 500 contributors from 25+ organizations on the way to this release.
The architecture is worth being clear about: llm-d is not a model server. It doesn’t replace vLLM or SGLang — it sits in front of them, distributing requests across a fleet of inference workers on Kubernetes. It integrates with the Kubernetes Gateway API Inference Extensions for endpoint selection. If you’re familiar with service meshes, think of it as purpose-built for the inference access patterns that general-purpose infrastructure handles poorly.
The 1.0 features that matter most for agentic workloads are on the Day 2 operations side. Multi-tenant quality of service uses priority- and fairness-based flow control to prevent latency-sensitive requests from being crowded out by agent traffic spikes. Session-aware routing ensures multi-turn conversations land near their cached KV state, reducing recomputation across agent turns. The router now has a hot standby with rolling updates, so upgrades don’t require taking inference capacity offline.
IBM Research reported running a quantum error correction workload at about $0.11 per million tokens blended, with a 98% prefix-cache hit rate. A separate benchmark with a ~753B-parameter model hit 6.6 million output tokens per minute at peak, serving 3,000 concurrent coding agents with zero preemptions. These are vendor-reported numbers from demos, not independent benchmarks, but the order of magnitude is credible for what prefix-cache sharing buys you in a multi-agent workload.
The project claims 5–10x cost reductions for context-heavy agentic workloads compared to commercial API pricing at current cloud rental rates. That claim is highly workload-dependent — prefix cache hit rate is the variable that matters most, and hit rate depends heavily on whether your agents are actually issuing similar prefixes. For coding agents operating on the same codebase, the gains can be substantial. For general-purpose chat, much less so.
What makes llm-d interesting now rather than just an interesting project is the 1.0 stable API commitment. Running pre-1.0 infrastructure tooling in production means absorbing breaking changes. The CNCF governance structure and the corporate backing mean there’s an actual support and stability story here, which is what a lot of organizations need before they’ll run open-source inference infrastructure at scale.
The project is Apache 2.0 and hosted under CNCF governance. Google is integrating it into GKE, which will be the fastest path to trying it for most teams.