Skip to content
Docs

Routing Modes

The routing mode is the single biggest lever on how a deployment feels under load. It is set on the Configuration page and applies fleet-wide, taking effect immediately.

Mode Per-server limit Queue Use when
Superfast clamped to 4 concurrent on Interactive chat and agents. Minimum time-to-first-token.
Hybrid 4 while quiet, the configured limit once busy on Mixed traffic. Fast when idle, high throughput under bursts.
Max concurrency (default) the configured limit (default 64) on Most deployments. A server’s full capacity: maximum aggregate throughput.
Queue all the configured limit on, deep Never reject. Bursts wait their turn.

A fresh install and an upgrade with no mode set both land on Max concurrency (since 2.2.60). Superfast holds every server to 4 requests at once, which keeps time-to-first-token minimal for a handful of users and queues everything beyond that: on a 96 GB GPU it held a 30B model to 1,446 tokens per second at 64 concurrent requests, against 5,812 at the server’s own limit. Choose Superfast or Hybrid when the first token matters more than total throughput.

Counter-intuitive but central. An engine given many concurrent requests interleaves them, so every request’s first token arrives later even though total throughput is higher. Capping concurrency per server keeps time-to-first-token low at the cost of aggregate tokens per second.

That is the whole trade, and it is why “everything feels slow” is more often a mode question than a hardware question — in both directions.

Hybrid switches per model pool, not globally, so a busy model does not drag a quiet one into throughput mode.

It goes busy when either in-flight requests or request rate crosses a threshold, and the decision is sticky: both signals must fall well below the threshold before it goes quiet again. Without that deadband a pool sitting near the threshold would oscillate, changing behaviour request to request.

The Configuration page polls live and shows per-pool in-flight, rate and hot state, so you can watch the switch happen rather than infer it.

Forces the waiter queue on with a deep ceiling and a long time-to-live, and serves strictly first-in-first-out. Requests wait instead of being rejected.

Use it for batch work where a slow answer beats a failed one. It does not create capacity — a sustained overload still ends in timeouts, just later. See Capacity and queueing.

503 and 504 bodies report the routing mode and the effective per-backend limit alongside the configured one, so “rejected at 4 because superfast is on” is distinguishable from “rejected at 32 because the pool is genuinely full” without cross-referencing anything.

The Dashboard’s load-balancer card names the active mode in its header for the same reason.