Prefill/Decode Disaggregation

A request to a language model has two very different phases. Prefill reads the whole prompt at once and is limited by compute. Decode emits one token at a time and is limited by memory bandwidth. Running both on the same GPU means neither phase gets hardware suited to it.

Disaggregation splits them: a prefill half computes the prompt’s cached state and hands it to a decode half, which generates the answer. The hand-off happens over a transfer engine — fastest over RDMA, and working without it.

Configuring a half

Servers → Create Server → Advanced → Disaggregation role: none, prefill or decode. The transfer backend, bootstrap port and network device are configured alongside it.

A pair is two servers on the same model, usually one per GPU.

A half is excluded from ordinary routing, deliberately

This is the most important thing to understand about the feature.

A prefill half has no decode loop; a decode half waits for cached state only its partner can send. Sent an ordinary request, neither returns an error — the decode half simply waits forever for a transfer that is never coming. The client hangs.

A hang is worse than a refusal: it occupies a slot, reads as slow hardware, and gives you nothing to act on. So Lifeboat removes disaggregated halves from normal routing before any other consideration, and a pool containing only halves returns a specific error naming that, rather than a generic “no backend available”.

The practical consequence: configuring the roles does not by itself give you a working endpoint. Something has to address the pair, and that is not the ordinary proxy.

What Lifeboat does and does not do

Configure both halves, including every engine setting	Yes
Size each half’s memory correctly, including when its partner is stopped	Yes
Keep halves out of ordinary routing so clients never hang	Yes
Report clearly when the transfer backend is missing	Yes
Pair a request across the two halves	No — see below

Pairing is small — pick a shared identifier, send the same request to both halves carrying it — and a working reference implementation ships in the repository under qa/pd-demo/, measured end to end. It is not in the control plane, because the proxy’s job is to protect clients from the hang described above, and the thing that pairs requests is a different front end that knows to address the pair.

Memory sizing

Both halves share a model path but are designed to run at the same time, so Lifeboat reserves memory for each on behalf of the other rather than treating them as alternative configurations of one model. A half you have stopped reserves nothing, exactly as any other stopped server.

A measured footprint is never shared across the two roles either: a prefill half and a decode half hold genuinely different amounts.

The transfer backend must be installed

The engine needs a KV transfer library (mooncake or nixl) and these are not in the standard image. Without one, a half used to load the model and then die with a stack trace pointing at a third-party build guide; Lifeboat now refuses the launch up front and names what is missing.

Over an ordinary network the transfer runs on TCP. On a host with an RDMA adapter the same configuration uses it, with no change on your side.

Whether disaggregation is faster depends on the link between the halves. If sending the cached state costs more than recomputing it, the split loses — so measure it on your own hardware before committing to the topology. No speed-up figure is published here because none has been measured on hardware that has the interconnect this feature exists for.

Sources and references

• Prefill and decode have different bottlenecks; separating them is an established technique in large-scale serving.
• Lifeboat’s routing exclusion, role configuration and memory sizing are implemented in the control plane and covered by its test suite.