Routing Modes
The routing mode is the single biggest lever on how a deployment feels under load. It is set on the Configuration page and applies fleet-wide, taking effect immediately.
The four modes
Section titled “The four modes”| Mode | Per-server limit | Queue | Use when |
|---|---|---|---|
| Superfast | clamped to 4 concurrent | on | Interactive chat and agents. Minimum time-to-first-token. |
| Hybrid | 4 while quiet, the configured limit once busy | on | Mixed traffic. Fast when idle, high throughput under bursts. |
| Max concurrency (default) | the configured limit (default 64) | on | Most deployments. A server’s full capacity: maximum aggregate throughput. |
| Queue all | the configured limit | on, deep | Never reject. Bursts wait their turn. |
A fresh install and an upgrade with no mode set both land on Max concurrency (since 2.2.60). Superfast holds every server to 4 requests at once, which keeps time-to-first-token minimal for a handful of users and queues everything beyond that: on a 96 GB GPU it held a 30B model to 1,446 tokens per second at 64 concurrent requests, against 5,812 at the server’s own limit. Choose Superfast or Hybrid when the first token matters more than total throughput.
Why a low limit makes things faster
Section titled “Why a low limit makes things faster”Counter-intuitive but central. An engine given many concurrent requests interleaves them, so every request’s first token arrives later even though total throughput is higher. Capping concurrency per server keeps time-to-first-token low at the cost of aggregate tokens per second.
That is the whole trade, and it is why “everything feels slow” is more often a mode question than a hardware question — in both directions.
Hybrid, and why it does not flap
Section titled “Hybrid, and why it does not flap”Hybrid switches per model pool, not globally, so a busy model does not drag a quiet one into throughput mode.
It goes busy when either in-flight requests or request rate crosses a threshold, and the decision is sticky: both signals must fall well below the threshold before it goes quiet again. Without that deadband a pool sitting near the threshold would oscillate, changing behaviour request to request.
The Configuration page polls live and shows per-pool in-flight, rate and hot state, so you can watch the switch happen rather than infer it.
Queue-all
Section titled “Queue-all”Forces the waiter queue on with a deep ceiling and a long time-to-live, and serves strictly first-in-first-out. Requests wait instead of being rejected.
Use it for batch work where a slow answer beats a failed one. It does not create capacity — a sustained overload still ends in timeouts, just later. See Capacity and queueing.
Diagnosing against the mode
Section titled “Diagnosing against the mode”503 and 504 bodies report the routing mode and the effective per-backend limit alongside the configured one, so “rejected at 4 because superfast is on” is distinguishable from “rejected at 32 because the pool is genuinely full” without cross-referencing anything.
The Dashboard’s load-balancer card names the active mode in its header for the same reason.
Sources and references
Section titled “Sources and references”- Rejections: Capacity and queueing
- Picking a backend: Load-balancing algorithms
- Setting it: Configuration reference