Troubleshooting — Slow Responses

Check these in order. The first two explain most cases and cost nothing to check.

1. The routing mode

Superfast caps each server at 4 concurrent requests. That is excellent for time-to-first-token and limiting for aggregate throughput. If you are running batch work on the default mode, that is your answer.

Switch to hybrid (fast when quiet, high throughput when busy) or max concurrency. See Routing modes.

A 503 body reports the effective limit alongside the configured one, which makes this immediately visible.

2. Queue wait

X-Lifeboat-Wait-ms on the response, or the Dashboard’s load-balancer card. A large wait means requests are queuing — a capacity problem, not a tuning one. Add a replica or raise concurrency.

3. GPU utilisation

Near zero while requests are slow means the bottleneck is not compute — look at queueing, or at prompt processing.

Near 100% means you are compute-bound and need either more GPU or a smaller model.

4. The model and its format

Decode is memory-bandwidth bound, so bytes-per-token sets the ceiling. A 4-bit quantization reads half the bytes of an 8-bit one and is usually about twice as fast.

On a memory-constrained machine the quantized build is faster and smaller. See Model formats.

5. Speculative decoding

2–4x on single-stream responses with identical output. Set Algorithm: Auto — it resolves to the best method the model’s weights allow, and for recognized models the draft model has already been fetched for you.

It does not help aggregate throughput under heavy load. See Speculative decoding.

Slow on a GGUF server specifically

Check max concurrency is 16 or below. That engine has fixed slots; above them the surplus queues inside the engine, where Lifeboat cannot see it — so the console reports no wait while requests are waiting, and time-to-first-token climbs steadily.

This is the single most common “the GPU must be saturated” misdiagnosis on that engine. See Capacity and queueing.

Every turn of a long conversation is slow

Some model families cannot reuse a cached prefix, so each turn reprocesses the whole conversation rather than just the new part.

For an agent resending a long transcript every turn this dominates everything else — a 35,000-token context can mean many seconds of pure recomputation per tool call. If long multi-turn work is your use case, prefer a model family that caches prefixes, and use the sticky session algorithm so a conversation stays on the backend that already has its prefix warm.

Time-to-first-token is bad but generation is fine

Prefill is the compute-bound half. A low-core host, a very long prompt, or a cold cache all show up here first. Smaller models improve it disproportionately.