Clusters & Load Balancing
This page is the summary. Each section below has a page of its own:
Beyond one machine, Lifeboat routes inference across many servers. This page covers the fleet (nodes and clusters) and the router in front of them, including what to do about every rejection it can return.
A node is another GPU host running the same image in agent mode. There is no self-registration and no gossip: the control plane is the single source of truth, and you add a node explicitly with its hostname, address and agent port. The control plane then polls it every 30 seconds to refresh its status and GPU inventory.
Authentication is a shared secret (LIFEBOAT_CLUSTER_SECRET) that must match on both sides. A mismatch surfaces as a 401 naming the node, rather than as a node that silently never appears.
Nothing is synchronized to a node in the background. Model weights are replicated when you ask for it; configuration, the registry, licensing and routing all stay in the control plane.
A node shows offline
Section titled “A node shows offline”- The agent is not running, or its port is not reachable from the control plane.
- The cluster secret does not match.
- If the agent is up but every server start on it fails, check that the agent container has a writable home directory — see Deployment.
An offline node raises a critical alert, and the alert resolves itself when the node returns.
Clusters
Section titled “Clusters”A cluster is a named pool of GPUs across nodes, aligned to a workload rather than to hardware — interactive, batch, one per tenant. Deploying a model to a cluster creates one server per member, sharing the model, port and settings, and each is started individually.
You can change a cluster’s load-balancing algorithm while servers are attached — it is a router setting. Changing its membership (GPUs, weights, names) requires detaching the servers first, because that changes the deploy shape.
Routing modes
Section titled “Routing modes”Set on the Configuration page; applies fleet-wide. This is the single biggest lever on how the deployment feels under load.
| Mode | Per-server limit | Queue | Use when |
|---|---|---|---|
| Superfast | 4 concurrent | on | Interactive chat and agents. Minimum time-to-first-token. |
| Hybrid | 4 while quiet, the configured limit once busy | on | Mixed traffic. Fast when idle, high-throughput under bursts. |
| Max concurrency (default) | the configured limit | on | Most deployments. Maximum aggregate throughput. |
| Queue all | the configured limit | on | Never reject. Bursts wait instead of failing. |
Hybrid switches per model pool, and is sticky: once a pool goes busy, both its in-flight count and its request rate must fall well below the threshold before it goes quiet again, so it does not flap at the boundary. The Configuration page shows the switch happening live.
A fresh install and an upgrade with no mode set both land on Max concurrency (since 2.2.60), so each server runs at its full limit. Pick Superfast or Hybrid when time-to-first-token for a few users matters more than total throughput.
Algorithms
Section titled “Algorithms”Set per cluster.
| Algorithm | Picks | Use when |
|---|---|---|
| Weighted round-robin (default) | in proportion to weight | Requests are of similar size. |
| Least connections | the least busy server, relative to weight | Prompt sizes vary a lot. Absorbs long-context outliers naturally. |
| Sticky session | the same server for the same client | Multi-turn chat — keeps that conversation’s cached prefix warm, which is a large latency win. |
Sticky affinity comes from an X-Lifeboat-Session-Id header if the client sends one, and otherwise from a digest of the API key, so conversational clients get prefix reuse without any wiring. The response reports only the digest, never the key. If the chosen server is unhealthy, at capacity or draining, the request falls through to least-connections rather than failing.
If a request’s candidates span more than one cluster, routing falls back to weighted round-robin — there is no principled way to combine two clusters’ algorithms. Use routing tiers to keep such workloads in separate pools.
Per-server knobs
Section titled “Per-server knobs”| Setting | Default | Effect |
|---|---|---|
| Weight | 1.0 | Share of traffic within the pool. 0 drains the server — it stops receiving traffic without being stopped. |
| Max concurrency | 32 | In-flight ceiling. 0 disables the gate. |
| Max input tokens | 0 (unbounded) | Requests estimated larger than this skip this server. |
| Routing tier | empty | A label that carves out an isolated pool. Same model, same cluster, separate traffic. |
On a GGUF server keep max concurrency at 16 or below. That engine has a fixed number of slots; above it, the surplus queues inside the engine where Lifeboat cannot see or report it, so the console shows no wait while requests are waiting.
Reading the routing headers
Section titled “Reading the routing headers”Every /v1/* response carries them, and they turn “is my traffic spread properly” into an observation:
| Header | Meaning |
|---|---|
X-Lifeboat-Server |
which backend served it |
X-Lifeboat-Pool |
the pool it was routed within |
X-Lifeboat-Weight |
that backend’s configured weight |
X-Lifeboat-Inflight |
current / maximum concurrency |
X-Lifeboat-Wait-ms |
queue wait; 0 means it went straight through |
X-Lifeboat-Algorithm |
the algorithm actually used |
X-Lifeboat-Reject-Reason |
on a rejection, why |
Diagnosing a rejection
Section titled “Diagnosing a rejection”503, no reject reason
Section titled “503, no reject reason”Every backend was at its concurrency ceiling and the queue is off. Raise max concurrency on the busy server, add a replica, or switch the routing mode to Queue all.
503 queue_full
Section titled “503 queue_full”The queue is on and at its depth ceiling. Same remedies — this is a capacity problem, not a queue-tuning one.
504 queue_timeout
Section titled “504 queue_timeout”Requests waited longer than the queue allows. The pool is sustaining more load than it can serve: add capacity. Raising the timeout only moves where it fails.
413 request_too_large
Section titled “413 request_too_large”No backend’s max input tokens covers the request. Raise it on the server that should take these, or shorten the prompt.
503 no_backends
Section titled “503 no_backends”Nothing is running for that model. Check the Servers page — the usual cause is a model name that does not match any running server.
501 no_embedding_backend / 503 embedding_only_pool
Section titled “501 no_embedding_backend / 503 embedding_only_pool”Direction mismatch. /v1/embeddings needs an embedding server; chat and completions need a generation server. An embedding engine physically cannot generate and a generation engine has no pooler, so these are refused rather than attempted — which is what stops an embeddings request being answered by a chat model half the time.
501 no_diffusion_backend
Section titled “501 no_diffusion_backend”An image, video or mesh endpoint with no diffusion server running. “Unsupported” and “supported but nothing started” are different problems, hence the distinct code.
Traffic share does not match the weights
Section titled “Traffic share does not match the weights”The Dashboard’s load-balancer card shows configured share against actual, with a drift badge. Drift beyond about 10% under steady load means a server is erroring (and being briefly benched), is at capacity, or is parked at weight 0.
Monitoring
Section titled “Monitoring”/api/lb/state returns a live snapshot — in-flight counts, dispatch and rejection counts, queue depth, wait percentiles, per-pool state and the active mode. /metrics exposes the same in Prometheus format, unauthenticated, for scraping.
Sources and references
Section titled “Sources and references”- Per-server settings in context: Servers and multi-GPU
- Setting the routing mode: Configuration and admin
- The endpoints being routed: API and automation