Skip to content
Docs

Context Length

Context Length is the per-request window. Auto uses the model’s native window, bounded by what the key-value cache can hold.

Two ceilings, and either can bind

Section titled “Two ceilings, and either can bind”

The effective limit is the smaller of:

  • the model’s own window — remedy: none, or extend it (below);
  • what the cache can hold — remedy: fewer concurrent requests, another GPU, or free the card.

The console reports which one binds, because the operator’s action differs completely. Both directions occur in practice: a deployment can have a cache 2.4x larger than the model’s declared window (so the model binds and the extra cache sits unused), and after extending the window the cache becomes the limit.

The advertised window is not the cache size

Section titled “The advertised window is not the cache size”

Worth stating plainly, because getting it wrong produces intermittent failures that look like a flaky backend.

The cache is capacity shared across all concurrent requests and is often far larger than any single request may use. What a client must stay under is the per-request limit. Lifeboat advertises that limit — not the pool — so a client that respects /v1/models cannot overflow.

A gateway that budgets prompts against a pool size will build a prompt that fits the pool, add a reserved completion on top, and get rejected mid-session with “exceeds the model’s maximum context length”. If you see that, check what your client believes the window is.

Extending beyond the native window

Section titled “Extending beyond the native window”

Ask for more than the model’s native window and Lifeboat generates a rope-scaling sidecar at launch. Two things to know before doing it:

  • It applies to every request, including short ones, and costs short-prompt accuracy. Extend by the smallest factor that clears your need, not the largest the model claims.
  • The slider defaults to native, never to its maximum, for exactly that reason. Extending is a deliberate act.

Some model families cannot be extended this way at all — a multimodal model using interleaved position encoding loses that encoding if rope scaling is applied. Lifeboat reports the envelope it can actually offer.

On the GGUF engine

Section titled “On the GGUF engine”

Different mechanics, and the trap is worth knowing: that engine has a fixed number of slots and the total context window divides across them. Ask for a 32,768-token window with 16 slots and each request gets 2,048.

Lifeboat treats context_length as the per-request window — what an operator means, and what the API advertises — and derives the total from it. Unset, it sizes from live free memory, the model file size and the model’s own trained window. If the per-request window would fall below a usable floor, it halves the slot count rather than serve a window that breaks every prompt.

Sources and references

Section titled “Sources and references”