Skip to content
Docs

Capacity & Queueing

Every request passes a capacity gate before it reaches a backend. This page is what happens at the limit and how to read the result.

The gate

Section titled “The gate”

Each backend has a max concurrency (default 32; 0 disables the gate). The effective limit also depends on the routing mode — superfast clamps it to 4.

When every candidate is at its limit, the request is either rejected immediately or parked on the queue, depending on the mode.

The queue

Section titled “The queue”

Off by default, forced on by queue-all mode, and available by environment variable in the others.

Knob Default Meaning
Time-to-live 30s (600s under queue-all) After this, the request gets 504.
Depth ceiling 4x the pool’s total concurrency (1000x under queue-all) Beyond it, 503 queue_full.
Deadline boost 10s Requests parked this long get priority over fresh ones, so nothing starves.

The queue redistributes; it does not create capacity. A sustained overload still fails, just later and in order.

Token-aware fit filtering

Section titled “Token-aware fit filtering”

Requests are sized before routing, and a backend whose max input tokens is smaller than the estimate is dropped from the candidate set. That keeps a large prompt away from a backend configured for short ones instead of letting it overflow deep in the engine.

Estimation reads the shape of the request body — messages, prompt, input, documents, and so on. Pre-tokenized integer input counts as one token per id. Audio uploads are sized by bytes, which under-estimates compressed formats; that only matters if a backend sets an input cap.

Every rejection

Section titled “Every rejection”
Status Reason What to do
503 (none) Capacity gate, queue off. Raise max concurrency, add a replica, or switch to queue-all.
503 queue_full Queue at its ceiling. Same remedies — a capacity problem.
504 queue_timeout Sustained more load than the pool can serve. Add capacity; raising the timeout moves the failure, not the cause.
413 request_too_large No backend’s input cap covers it. Raise the cap or shorten the prompt.
503 no_backends Nothing running for that model — usually a name matching nothing.
503 embedding_only_pool A generation request aimed at an embedding model.
501 no_embedding_backend /v1/embeddings with no embedding server.
501 no_diffusion_backend An image or video endpoint with no diffusion server.

503 and 504 bodies also report the routing mode and the effective per-backend limit.

The GGUF slot trap

Section titled “The GGUF slot trap”

Worth stating separately because it is invisible from the console.

The GGUF engine has a fixed number of slots. Above that, surplus requests queue inside the engine, where Lifeboat cannot see them — so X-Lifeboat-Wait-ms reports 0 and the load-balancer card shows an empty queue while requests are genuinely waiting, and time-to-first-token climbs.

Keep max concurrency at 16 or below on GGUF servers, so the router’s accounting is truthful and the surplus waits where you can see it.

Monitoring

Section titled “Monitoring”

GET /api/lb/state returns in-flight counts, dispatch and rejection counts, queue depth, wait percentiles and per-pool state. GET /metrics exposes the same in Prometheus format, unauthenticated. See Observability and metrics.

Sources and references

Section titled “Sources and references”