Distributed LLM inference has a coordination problem. The approaches that work well in a data center — shared memory, high-bandwidth interconnects, centralized orchestration — don’t translate cleanly to a collection of consumer machines spread across a home network. Petals tried p2p inference in 2022 with mixed results; Red Hat’s LLM-D targets Kubernetes clusters with a very different set of assumptions. What Mesh LLM, published yesterday on the iroh blog, proposes is different: treat inference as a peer-to-peer routing problem.

iroh is a Rust networking library that handles the unglamorous parts of p2p connectivity — hole-punching, NAT traversal, relay fallback — and presents each node as a QUIC endpoint identified by a public key. You don’t need a central discovery server; you need a public key and a way to tell your peer about it. Mesh LLM builds its inference infrastructure on top of this. Each participating machine runs an iroh endpoint, announces which models it has loaded, and can either serve requests directly or route them to better-placed peers.

The three serving modes are worth understanding:

  • Local: the requesting machine’s GPU handles the inference directly
  • Routed: the request is forwarded to a peer that already has the model loaded and ready
  • Split (via the “Skippy” engine): a model too large for any single machine is partitioned by layer range across multiple nodes — layers 0–15 on one, 16–31 on the next — with activations flowing sequentially through the pipeline

The protocol structure separates these concerns at the ALPN level: mesh-llm/1 handles gossip, routing, and HTTP proxying; mesh-llm-control/1 manages configuration sync and ownership; and skippy-stage/2 carries the latency-sensitive activation transport for split models. Within the main protocol, eight stream types identified by leading byte handle everything from peer announcements to NAT traversal requests.

What actually limits this

The crucial insight, surfaced quickly in the Hacker News thread, is that split inference’s bottleneck is latency, not bandwidth. Each inter-node transfer carries only a few kilobytes of activations — the tensor slice passing from one layer partition to the next. Raw bandwidth is fine; even a modest home internet connection can handle kilobytes per token. What compounds badly is round-trip latency. A model with 100 layers, split across several nodes, incurs that latency dozens of times per generated token. Add 50ms per hop across the internet and you’ve added seconds per token before accounting for compute.

In practice this means Mesh LLM is a local network tool wearing peer-to-peer clothing. The project reports around 16 tok/s for Qwen 235B split across two nodes on a LAN — genuinely usable — but that number degrades sharply the moment internet latency enters the picture. If your mesh consists of machines in the same building, this works. If it’s machines across different cities, it probably doesn’t.

The privacy situation in split mode is worth flagging: every node in the pipeline sees the intermediate activations flowing through its stage. This isn’t encryption-level sensitive for most use cases, but it means the model’s internal computations are visible to peers. The project explicitly recommends keeping meshes to trusted participants and warns against using it for sensitive inference. There’s also no described mechanism for detecting a peer that deliberately corrupts the activations it forwards, which matters if your trust model includes skeptical participants.

The useful case

Where this fits: not replacing cloud inference, but pooling existing hardware. If you have two machines each with 24 GB VRAM and want to run a 48B model that neither can hold alone, Mesh LLM is a plausible path to doing that without a Kubernetes cluster or a formal serving setup. The binary is about 18 MB, exposes an OpenAI-compatible endpoint at localhost:9337/v1, and supports 40+ models from 500M-parameter laptops weights up to 235B mixture-of-experts. Any tool that already speaks the OpenAI API protocol works with it immediately.

The iroh foundation matters here in ways that previous p2p inference experiments lacked. NAT traversal and hole-punching being handled at the library level means Mesh LLM doesn’t need to solve that problem itself. The QUIC transport provides reliability and congestion control without building a bespoke protocol. That’s a real difference from projects that roll their own p2p layer and spend half their effort on connectivity edge cases.

Whether the latency constraint turns out to be fundamental or engineering-solvable depends on what you’re trying to do. For pooling machines in the same room, it’s fine. For building a distributed inference network across the internet — the more ambitious vision — it’s a serious obstacle that no one has cleanly solved.