Load-Balancing Algorithms

The algorithm is set per cluster and decides which backend takes each request. It is independent of the routing mode, which decides how much each backend will take.

The four algorithms

Algorithm	Picks	Use when
Weighted round-robin (default)	in proportion to weight, smoothly	Requests are similar in size.
Least connections	the least busy backend, relative to weight	Prompt sizes vary a lot — absorbs long-context outliers naturally.
Sticky session	the same backend for the same client	Multi-turn chat.
KV-aware	the backend that already holds this prompt’s prefix	Many clients share one long prompt — a system prompt, a tool manifest, a retrieved document.

KV-aware routing

Sticky sessions keep one client on one backend. The case that dominates agent and retrieval traffic is different: many different clients sending the same long prefix. They share no session, so a session hash spreads them across the pool and every backend ends up processing the same opening tokens.

KV-aware routing decides from the prompt itself. The control plane remembers which backend was recently sent which prefix and prefers that one.

Measured on an RTX PRO 6000 Blackwell, two backends, nine distinct prompts against a cache that can hold about five:

	traffic split	cache reuse
Weighted round-robin	23 / 22	3.4%
KV-aware	25 / 20	48.3%

Fourteen times the reuse, with the load still evenly split — which is the point. Affinity is applied only among backends that are already among the least busy, so a backend holding a popular prefix is never allowed to become a hot spot. When no backend holds a useful match, the request is spread exactly as weighted round-robin would spread it, so the worst case is the default algorithm.

A stale guess costs nothing beyond the ordinary processing that would have happened anyway. It pays off most when the number of distinct long prompts is larger than one backend can cache but fits the pool once divided among them; if every prompt already fits everywhere, there is nothing to win and the simpler algorithms are fine.

It composes with KV cache tiering: routing decides where a prefix is reused, tiering decides how long it survives.

One client cannot spend another’s cache

What Lifeboat remembers about each backend is bounded, so without a rule the clients share it first-come-first-served — and a single client sending many distinct prompts evicts everyone else’s entries. Their next request then misses and reprocesses the prompt, paying for the noisy client’s traffic. That is precisely the situation KV-aware routing exists for, so it was the case that undermined it.

Each client is now guaranteed an equal share of what a backend holds, and only a client over its share is evicted first. A client working alone still gets the whole thing, so the guarantee costs nothing when there is no contention. The Dashboard’s KV-Aware Routing panel shows the split once more than one client is active.

Clients are identified by the X-Lifeboat-Session-Id header when sent, and otherwise by a one-way digest of the API key — the key itself is never stored or displayed.

Why sticky sessions are worth it for chat

A conversation resends its whole transcript every turn. If it lands on the backend that served the previous turn, that backend already has the prefix cached and only processes the new part. Land it somewhere else and the entire transcript is reprocessed.

On a long agent conversation that is the difference between a fast turn and a slow one, and it costs nothing to enable.

Affinity comes from an X-Lifeboat-Session-Id header if the client sends one, and otherwise from a digest of the API key — so conversational clients get the benefit with no wiring. The response reports only the digest, never the key.

If the chosen backend is unhealthy, at capacity or draining, the request falls through to least-connections rather than failing. Affinity is an optimization, not a constraint.

Per-server knobs

Setting	Default	Effect
Weight	1.0	Share of traffic in the pool. 0 drains — the backend stops receiving traffic without being stopped.
Max concurrency	32	In-flight ceiling. 0 disables the gate.
Max input tokens	0 (unbounded)	Requests estimated larger than this skip this backend.
Routing tier	empty	Isolates a sub-pool.

Weight 0 is the graceful-drain mechanism. Set it, let in-flight requests finish, then stop the server — no failed requests, no window where the pool is short.

The headers on every response

Header	Meaning
X-Lifeboat-Server	which backend served it
X-Lifeboat-Pool	the pool it was routed within
X-Lifeboat-Weight	that backend’s configured weight
X-Lifeboat-Inflight	current / maximum concurrency
X-Lifeboat-Wait-ms	queue wait; 0 means straight through
X-Lifeboat-Algorithm	the algorithm actually used
X-Lifeboat-Sticky-Key	the affinity digest, when sticky is in effect
X-Lifeboat-Prefix-Blocks	matched/total prefix blocks the chosen backend already held, when KV-aware is in effect
X-Lifeboat-Reject-Reason	on a rejection, why

Logging X-Lifeboat-Server and X-Lifeboat-Wait-ms client-side makes a later “the API was slow” answerable instead of speculative.

Verifying the weights

Drive steady load, then compare each backend’s dispatch share against its configured weight — the Dashboard’s load-balancer card does this for you with a drift badge.

Drift beyond roughly 10% under steady load means a backend is erroring (and being briefly benched), is at capacity, or is parked at weight 0.

Mixed clusters

If a request’s candidates span several clusters, or include a standalone server, routing falls back to weighted round-robin — combining two clusters’ algorithms has no principled answer. Use routing tiers to keep such workloads apart.

Sources and references

• Capacity and rejection: Capacity and queueing
• Behaviour under load: Routing modes
• Grouping: Clusters