Skip to content
Docs

Load-Balancing Algorithms

The algorithm is set per cluster and decides which backend takes each request. It is independent of the routing mode, which decides how much each backend will take.

The three algorithms

Section titled “The three algorithms”
Algorithm Picks Use when
Weighted round-robin (default) in proportion to weight, smoothly Requests are similar in size.
Least connections the least busy backend, relative to weight Prompt sizes vary a lot — absorbs long-context outliers naturally.
Sticky session the same backend for the same client Multi-turn chat.

Why sticky sessions are worth it for chat

Section titled “Why sticky sessions are worth it for chat”

A conversation resends its whole transcript every turn. If it lands on the backend that served the previous turn, that backend already has the prefix cached and only processes the new part. Land it somewhere else and the entire transcript is reprocessed.

On a long agent conversation that is the difference between a fast turn and a slow one, and it costs nothing to enable.

Affinity comes from an X-Lifeboat-Session-Id header if the client sends one, and otherwise from a digest of the API key — so conversational clients get the benefit with no wiring. The response reports only the digest, never the key.

If the chosen backend is unhealthy, at capacity or draining, the request falls through to least-connections rather than failing. Affinity is an optimization, not a constraint.

Per-server knobs

Section titled “Per-server knobs”
Setting Default Effect
Weight 1.0 Share of traffic in the pool. 0 drains — the backend stops receiving traffic without being stopped.
Max concurrency 32 In-flight ceiling. 0 disables the gate.
Max input tokens 0 (unbounded) Requests estimated larger than this skip this backend.
Routing tier empty Isolates a sub-pool.

Weight 0 is the graceful-drain mechanism. Set it, let in-flight requests finish, then stop the server — no failed requests, no window where the pool is short.

The headers on every response

Section titled “The headers on every response”
Header Meaning
X-Lifeboat-Server which backend served it
X-Lifeboat-Pool the pool it was routed within
X-Lifeboat-Weight that backend’s configured weight
X-Lifeboat-Inflight current / maximum concurrency
X-Lifeboat-Wait-ms queue wait; 0 means straight through
X-Lifeboat-Algorithm the algorithm actually used
X-Lifeboat-Sticky-Key the affinity digest, when sticky is in effect
X-Lifeboat-Reject-Reason on a rejection, why

Logging X-Lifeboat-Server and X-Lifeboat-Wait-ms client-side makes a later “the API was slow” answerable instead of speculative.

Verifying the weights

Section titled “Verifying the weights”

Drive steady load, then compare each backend’s dispatch share against its configured weight — the Dashboard’s load-balancer card does this for you with a drift badge.

Drift beyond roughly 10% under steady load means a backend is erroring (and being briefly benched), is at capacity, or is parked at weight 0.

Mixed clusters

Section titled “Mixed clusters”

If a request’s candidates span several clusters, or include a standalone server, routing falls back to weighted round-robin — combining two clusters’ algorithms has no principled answer. Use routing tiers to keep such workloads apart.

Sources and references

Section titled “Sources and references”