22:38
17d ago
→n0 team open-sources Mesh LLM, pools multiple machines GPUs into OpenAI-compatible inference network
The n0 team open-sourced Mesh LLM, which pools GPUs from multiple ordinary machines to run large models and exposes an OpenAI-compatible API at localhost:9337/v1. Requests can run locally, route to a peer with the model loaded, or split a model across machines in a layer-wise pipeline. Underneath, iroh handles NAT traversal and direct QUIC connections with no central server. Over 40 models are included, from 0.5B to 235B MoE. The post doesn't disclose inference latency or throughput numbers, so I'd hold off on production assumptions.
82
SCORE
H1·K1·R1