Skip to content
Docs

Memory & Sizing

The single most misunderstood setting in Lifeboat, so this page is precise about it.

Memory Fraction is a fraction of FREE memory

Section titled “Memory Fraction is a fraction of FREE memory”

Not of the card. The engine measures free memory at the moment the server starts, multiplies by this fraction, loads the weights, and gives the remainder to the key-value cache.

So 0.9 on a card with 40 GB free means 36 GB, regardless of whether the card is 48 GB or 96 GB. Memory that other processes already hold is already excluded, because the measurement happens after they allocated — adding it back double-counts.

Getting this backwards makes every co-residency decision wrong, usually in the direction of one server taking the whole card.

Leave it on Auto

Section titled “Leave it on Auto”

Auto derives the value from live GPU state at every start. It also:

  • Reserves room for other models that may start on the same GPU, so a generous first server does not take the card. The reserve is per model, not per server row, and is capped at 40% of the card.
  • Uses a measured footprint for the same weights when one exists, instead of a cold estimate. Start one variant of a model and every other row for the same weights sizes itself from that measurement.
  • Re-derives after a reboot, so a server sized on a busy GPU sizes itself correctly once the GPU is free.

A pinned number is a number that was right once. The classic failure: pin 0.9 while the GPU is empty, and that server takes the card on every subsequent start, so the second server never starts and looks like the one at fault.

Auto is stored as “unset”. If the page shows a number, the server is pinned — tick Auto to unpin it.

Why my second server will not start

Section titled “Why my second server will not start”

In order of likelihood:

  1. The first one took the card. The engine pre-allocates its cache up to whatever ceiling it is given, so a generous first server does not leave room later. Put both on Auto and restart them.
  2. Another process holds VRAM — a previous engine that did not exit, or a notebook. Check the Dashboard’s GPU panel.
  3. Too many server rows. The reserve scales with rows, not with what will actually run. Three stale rows can hold back a quarter of the card. Delete rows you do not use.
  4. It genuinely does not fit. Reduce context length, use a quantized checkpoint, or add a GPU.

Measured footprints

Section titled “Measured footprints”

After a server reaches running, Lifeboat measures what it actually holds and records it against (model path, tensor parallelism). Later starts of the same weights size from that number.

Two safeguards worth knowing, because they explain a footprint that is missing:

  • A measurement is discarded if another server started during the window, so a peer’s allocation is never attributed to this server. A footprint is never re-derived once recorded, so a wrong one would reproduce itself forever.
  • A launch that hit the clamp ceiling is not recorded: a server that starts last on a nearly-full GPU takes whatever is left, which measures permission rather than need.

Remote and GGUF servers

Section titled “Remote and GGUF servers”
  • Remote rows are excluded from local GPU accounting — a GPU index on a remote row refers to a card on another machine.
  • The GGUF engine does not use this setting at all. It offloads by layer count, and the control is disabled for those servers. Its sizing question is context, not fraction — see Context length.

Unified memory

Section titled “Unified memory”

On Apple Silicon, NVIDIA GB10 and Grace-Hopper the GPU shares system memory, and the card reports no separate total. Lifeboat reads available system memory instead, counting reclaimable page cache as available — otherwise a host that is three-quarters idle looks full.

Sources and references

Section titled “Sources and references”