LLM Router
One OpenAI-compatible endpoint in front of all your inference engines.
LLM Router is a fast, self-hosted reverse proxy that sits in front of every OpenAI-compatible engine you run — LM Studio, oMLX, vLLM, llama.cpp, Strata, and anything else — and exposes a single OpenAI-compatible API. It is not an inference engine or a cluster: it is the smart layer between your apps and your hardware, handling routing, priorities, load balancing, private connectivity, monitoring, and administration. It is built for the messy reality of self-hosted inference: your PCs shut down, engines unload models when idle, one box is fast but small and another is slow but huge. The router prefers warm models, fails over before the first byte, and never assumes a static datacenter behind it.
What it does
- One API, many engines. Chat completions, the stateful Responses API, legacy text completions, embeddings, images, audio, moderations, and a model catalog — all through a single OpenAI-compatible endpoint.
- Smart routing. A priority queue, load balancing, warm-model preference, and failover before the first byte, so a cold or overloaded engine never stalls your request.
- Private by default. A reverse tunnel lets an inference host connect without exposing any public port — ideal for a home or LAN box behind NAT.
- Share and pool. Virtual models combine engines across machines into one requestable id, with per-model add, remove, and reorder control.
- Ownership and privacy. Per-server and per-model share scopes, a privacy mode that hides your activity from admins, and analytics scoped to what each user owns.
- Monitoring and analytics. Live server status, model discovery, and usage broken down by user, key, server, and model.
Inside the router
Screenshots coming soon — the boxes below describe what each image will show.