Skip to content
Docs

Speculative Decoding

A small draft model proposes several tokens and the real model verifies them in one pass. Output is identical to decoding without it — this is a latency optimization, not an approximation.

What it does and does not help

Section titled “What it does and does not help”

Helps: interactive, single-stream responses — chat, coding assistants, anything where one person waits. 2–4x typical.

Does not help: aggregate throughput under heavy concurrency. When the GPU is already saturated by many requests there is no idle capacity for the draft model to use, and it can cost a little.

So it is a latency lever for interactive deployments, and roughly neutral on batch ones.

Algorithm: Auto

Section titled “Algorithm: Auto”

The right answer for almost everyone. Auto resolves at start to the best method the model’s own weights allow:

  1. A verified draft head for that model, if one is known and has been fetched.
  2. The model’s built-in prediction layers, if it ships them.
  3. Otherwise an n-gram method, which needs no extra model and works on anything.

So a new server on any servable model gets the fastest method its weights allow with no setup. None is the explicit opt-out.

Draft models are fetched for you

Section titled “Draft models are fetched for you”

When a recognized model finishes downloading, Lifeboat fetches the matching draft head alongside it and records it on the model row. Nothing is required from you, and a failed fetch leaves the base model fully usable without speculative decoding — it never blocks.

Air-gapped deployments have nothing to fetch from; stage the head and set the path.

Heads are algorithm-specific

Section titled “Heads are algorithm-specific”

A head trained for one algorithm loads partway with another and then fails deep in the engine with an error naming neither. Lifeboat refuses that combination up front with a clear message, and the console re-fills the correct head when you change the algorithm.

If you set a head path by hand, make sure it matches the algorithm.

Requirements per algorithm

Section titled “Requirements per algorithm”
Algorithm Needs
Draft-head methods A trained head matched to the base model. Enforced at start, not at create — so a row can be created and the path filled in later.
Built-in prediction Nothing; uses the model’s own layers.
N-gram Nothing at all.

Interactions

Section titled “Interactions”
  • It is mutually exclusive with the overlap scheduler, which Lifeboat disables automatically when you enable it.
  • Embedding servers strip it entirely — there is no decode loop to speculate on.
  • Not available on the GGUF engine. Lifeboat maps these settings to the tensor engine only, so a GGUF server runs standard decoding. On hardware where GGUF is the right choice anyway this matters less than it sounds: the quantized path is already several times faster there.
  • Not validated on remote nodes; those fall back to standard decoding.

Sources and references

Section titled “Sources and references”