Skip to content
Docs

Routing Modes

The routing mode is the single biggest lever on how a deployment feels under load. It is set on the Configuration page and applies fleet-wide, taking effect immediately.

The four modes

Section titled “The four modes”
Mode Per-server limit Queue Use when
Superfast (default) clamped to 4 concurrent off Interactive chat and agents. Minimum time-to-first-token.
Hybrid 4 while quiet, the configured limit once busy off Mixed traffic. Fast when idle, high throughput under bursts.
Max concurrency the configured limit (default 32) off Batch. Maximum aggregate throughput.
Queue all the configured limit on, deep Never reject. Bursts wait their turn.

A fresh install and an upgrade both land on Superfast, deliberately: it is the right default for interactive use, which is what most deployments do first.

Why a low limit makes things faster

Section titled “Why a low limit makes things faster”

Counter-intuitive but central. An engine given many concurrent requests interleaves them, so every request’s first token arrives later even though total throughput is higher. Capping concurrency per server keeps time-to-first-token low at the cost of aggregate tokens per second.

That is the whole trade, and it is why “everything feels slow” is more often a mode question than a hardware question — in both directions.

Hybrid, and why it does not flap

Section titled “Hybrid, and why it does not flap”

Hybrid switches per model pool, not globally, so a busy model does not drag a quiet one into throughput mode.

It goes busy when either in-flight requests or request rate crosses a threshold, and the decision is sticky: both signals must fall well below the threshold before it goes quiet again. Without that deadband a pool sitting near the threshold would oscillate, changing behaviour request to request.

The Configuration page polls live and shows per-pool in-flight, rate and hot state, so you can watch the switch happen rather than infer it.

Queue-all

Section titled “Queue-all”

Forces the waiter queue on with a deep ceiling and a long time-to-live, and serves strictly first-in-first-out. Requests wait instead of being rejected.

Use it for batch work where a slow answer beats a failed one. It does not create capacity — a sustained overload still ends in timeouts, just later. See Capacity and queueing.

Diagnosing against the mode

Section titled “Diagnosing against the mode”

503 and 504 bodies report the routing mode and the effective per-backend limit alongside the configured one, so “rejected at 4 because superfast is on” is distinguishable from “rejected at 32 because the pool is genuinely full” without cross-referencing anything.

The Dashboard’s load-balancer card names the active mode in its header for the same reason.

Sources and references

Section titled “Sources and references”