Skip to content
Docs

Memory & Sizing

The single most misunderstood setting in Lifeboat, so this page is precise about it.

Memory Fraction is a fraction of FREE memory

Section titled “Memory Fraction is a fraction of FREE memory”

Not of the card. The engine measures free memory at the moment the server starts, multiplies by this fraction, loads the weights, and gives the remainder to the key-value cache.

So 0.9 on a card with 40 GB free means 36 GB, regardless of whether the card is 48 GB or 96 GB. Memory that other processes already hold is already excluded, because the measurement happens after they allocated — adding it back double-counts.

Getting this backwards makes every co-residency decision wrong, usually in the direction of one server taking the whole card.

Leave it on Auto

Section titled “Leave it on Auto”

Auto derives the value from live GPU state at every start. It also:

  • Reserves room for other models that may start on the same GPU, so a generous first server does not take the card. The reserve is per model, not per server row, is capped at 40% of the card, and counts only servers that could actually start — a stopped server reserves nothing.

    Prefill and decode halves of a disaggregated pair are the exception to “per model”: they share a model path but are designed to run at the same time, so each reserves for the other rather than being treated as alternatives.

  • Uses a measured footprint for the same weights when one exists, instead of a cold estimate. Start one variant of a model and every other row for the same weights sizes itself from that measurement.

  • Re-derives after a reboot, so a server sized on a busy GPU sizes itself correctly once the GPU is free.

A pinned number is a number that was right once. The classic failure: pin 0.9 while the GPU is empty, and that server takes the card on every subsequent start, so the second server never starts and looks like the one at fault.

Auto is stored as “unset”. If the page shows a number, the server is pinned — tick Auto to unpin it.

Why my second server will not start

Section titled “Why my second server will not start”

In order of likelihood:

  1. The first one took the card. The engine pre-allocates its cache up to whatever ceiling it is given, so a generous first server does not leave room later. Put both on Auto and restart them.

  2. Another process holds VRAM — a previous engine that did not exit, or a notebook. Check the Dashboard’s GPU panel.

  3. Too many server rows you have never started. The reserve scales with rows, not with what will actually run, so several never-started duplicates can hold back a quarter of the card. Delete the ones you do not intend to use.

    A server you stopped is not one of them. Stopping is an explicit “off”, and Lifeboat treats it as one: a stopped server reserves nothing and its memory is free immediately, so you can start another model on that GPU without deleting anything. A server in error does still reserve, because the automatic restart can bring it back — unless the restart is permanently blocked, in which case it does not.

  4. It genuinely does not fit. Reduce context length, use a quantized checkpoint, or add a GPU.

Measured footprints

Section titled “Measured footprints”

After a server reaches running, Lifeboat measures what it actually holds and records it against (model path, tensor parallelism). Later starts of the same weights size from that number.

Two safeguards worth knowing, because they explain a footprint that is missing:

  • A measurement is discarded if another server started during the window, so a peer’s allocation is never attributed to this server. A footprint is never re-derived once recorded, so a wrong one would reproduce itself forever.
  • A launch that hit the clamp ceiling is not recorded: a server that starts last on a nearly-full GPU takes whatever is left, which measures permission rather than need.

Remote and GGUF servers

Section titled “Remote and GGUF servers”
  • Remote rows are excluded from local GPU accounting — a GPU index on a remote row refers to a card on another machine.
  • The GGUF engine does not use this setting at all. It offloads by layer count, and the control is disabled for those servers. Its sizing question is context, not fraction — see Context length.

Unified memory

Section titled “Unified memory”

On Apple Silicon, NVIDIA GB10 and Grace-Hopper the GPU shares system memory, and the card reports no separate total. Lifeboat reads available system memory instead, counting reclaimable page cache as available — otherwise a host that is three-quarters idle looks full.

Sources and references

Section titled “Sources and references”