Servers & Multi-GPU

This page is the summary. Each section below has a page of its own:

An inference server is one model running on one or more GPUs. This page is the operator reference: how to create one, what each setting does, and what to do about every state and error message the page can show.

Create and start a server

Servers → Create Server.
Model — any registered model. If the weights are not downloaded yet, the server can still be created; it resolves the path at start.
Name — used in logs, alerts and the routing headers. It does not have to match the model.
Port — must be free on the host. A clash is reported as port_in_use rather than left to fail.
Leave Tensor Parallel Size, Memory Fraction and Context Length on Auto unless you have a reason not to. Auto is not a placeholder: it re-derives from live GPU state at every start, so a server sized on a busy GPU sizes itself correctly once the GPU is free.
Create, then Start.

Starting is not instant. The engine loads weights and then captures CUDA graphs, which on a large mixture-of-experts model is legitimately several minutes and logs a line per captured shape. The status stays starting throughout; that is not a hang.

What each status means

Status	Meaning	What to do
`starting`	The process is up and loading.	Watch the log. A server is only treated as stuck after 15 minutes and 5 minutes with nothing written to its log — silence, not elapsed time, is the signal.
`running`	Healthy and in the routing pool.	—
`stopped`	Deliberately stopped. Not restarted on boot.	Start it when wanted.
`error`	Failed to start, or crashed and exhausted its retries.	Read the message on the row — it names the category, not just “failed”. Fix and start again.

On a host reboot Lifeboat restarts the servers that were up (running or starting). stopped is an explicit “off”, and error rows are deliberately not relaunched — with several server rows per model, relaunching every row that ever failed once is how a GPU ends up full of servers nobody started. A crash during a session is still recovered automatically by the restart loop.

Why a server will not start

The message on the row is classified, not generic. Each of these has a different fix.

Out of memory

GPU ran out of memory while loading the model.

The GPU had less free memory than the model needed. In order of likelihood:

Another process is holding VRAM. Check the Dashboard’s GPU panel. A previous engine that did not exit, or a notebook, is the usual culprit.
Too many server rows. Lifeboat holds memory back for other servers that might start, capped at 40% of the card. Several stale or duplicate rows for the same model therefore cost real headroom — delete rows you do not use.
The model genuinely does not fit. Reduce Context Length, use a quantized checkpoint, or add a GPU and raise tensor parallelism.

If a server starts but a later one cannot, that is usually the first one having taken the card rather than the second being too large — see Memory Fraction below.

`insufficient_shared_memory`

Only with tensor parallelism above 1. The GPUs communicate through /dev/shm, and the container’s default is too small. Raise it (LIFEBOAT_SHM_SIZE, or shmSize in Helm) and restart the container — it is fixed at container creation and cannot be changed at runtime. Lifeboat checks this before launching, so you get this message instead of an unexplained “NCCL unhandled system error”.

Because it cannot be fixed without recreating the container, automatic restarts are disabled for this failure rather than retried pointlessly.

`gpu_incompatible`

The GPU is older than the engine supports (below compute capability 8.0). Not fixable by configuration. Retries are disabled.

`model_too_large`

Refused before launching, based on live free VRAM rather than the card’s nameplate capacity. Free memory on the GPU, reduce context length, or use a smaller or quantized model. Note this check is skipped for GGUF servers, which stream layers to the GPU and can legitimately run a model larger than VRAM.

`unsupported_model`

The model cannot be served in the form it is in. The most common causes:

A sharded GGUF. Merge it first (gguf-split --merge); the engine requires a single file.
A GGUF whose architecture the tensor engine cannot map. Lifeboat routes these to the GGUF engine automatically; this error means neither engine can serve it.
Speculative decoding configured without a draft model for an algorithm that requires one.

A reasoning or tool-call parser the model cannot support

a reasoning parser is configured, but this model’s tokenizer does not encode the think-end token as a single token

Clear Reasoning Parser and Tool Call Parser in the server’s Advanced settings. These only shape generated text, so they are never correct for an embedding or reranker model — which is the case this most often appears in.

`invalid_extra_args`

Something in Extra Arguments is malformed. The field is split like a shell command line, so JSON must be in single quotes — otherwise the double quotes are stripped and the engine fails with a bare JSON parse error naming nothing useful.

`llamacpp_load_failed`

The GGUF engine could not load the file: a corrupt or unsupported quantization, a sharded set missing shards, or — for a vision model — a missing or mismatched projector. The server log carries the engine’s own error line.

`llamacpp_remote_unsupported`

GGUF servers run on the control-plane node only in this release. Deploy it locally.

`port_in_use`

Something else holds the port. Pick another, or stop whatever holds it.

Tensor Parallel Size

How many GPUs one model is split across: 1, 2, 4 or 8. Auto picks the smallest power of two that fits comfortably, preferring 1.

More GPUs let a larger model fit and add memory bandwidth, but add communication overhead — so TP 2 is not twice as fast.
Needs /dev/shm sized for it (above).
Ignored by the GGUF engine, which splits layers across whatever GPUs it can see.
On a remote node, Auto cannot inspect the box and uses 1.

Memory Fraction

The single most misunderstood setting, so be precise about it: it is a fraction of the GPU’s FREE memory at the moment the server starts, not of the card’s total. The engine measures free memory, multiplies by this fraction, loads the weights, and gives the remainder to the key-value cache. So 0.9 on a card with 40 GB free means 36 GB, regardless of how large the card is.

Leave it on Auto. Auto derives the value from live GPU state at every start, reserves room for other models on the same GPU, and uses a previously measured footprint for the same weights when it has one. A pinned number is a number that was right once: pin 0.9 on an empty GPU and that server will take the card every time, and the next server will not start.

Two consequences worth knowing:

Auto is stored as “unset”. If the page shows a number, the server is pinned — tick Auto to unpin it. (A server deployed to a cluster before 2.2.44 could be pinned without anyone choosing that; re-tick Auto on those.)
The GGUF engine does not use this at all — it offloads by layer count — and the control disables itself for those servers.

Context Length

The per-request window. Auto uses the model’s native window, bounded by what the key-value cache can actually hold.

The effective limit is the smaller of the model’s window and the cache, and the page reports both. A model advertising 262,144 tokens on a deployment whose cache holds 100,000 is a 100,000-token server; raising the setting cannot change that. Ask for more than the native window and Lifeboat extends it with a rope-scaling sidecar — but static extension applies to every request and costs short-prompt accuracy, so raise it deliberately rather than to the maximum.

Speculative Decoding

A draft model proposes tokens and the real model verifies them, giving a 2–4x single-stream speedup with identical output. It helps interactive use and does nothing for aggregate throughput under load.

Algorithm: Auto is the right answer for almost everyone. It resolves at start to the best method the model’s own weights allow: a matching draft head if one is known and has been fetched, the model’s built-in prediction layers if it has them, and otherwise an n-gram method that needs no extra model at all. None is the explicit opt-out.

For recognized models Lifeboat fetches the draft head for you when the model finishes downloading, so this is usually zero-configuration.
Draft heads are algorithm-specific. A head trained for one algorithm loads partway with another and then fails deep in the engine; Lifeboat refuses that combination up front with a clear message. Changing the Algorithm dropdown re-fills the matching head.
Not validated on remote nodes; those fall back to standard decoding.

Embedding and reranker models

You do not configure this. Lifeboat detects an embedding model at start — preferring the pooling configuration on disk over the model’s name, because a decoder-based embedder is byte-identical to its chat counterpart in config.json — sets the flag, strips every generation-only setting, and records what it did in the audit log.

Two things follow. Do not set a reasoning or tool-call parser on one (see above). And be aware that a generation request sent to an embedding server kills the engine process: Lifeboat blocks every path it controls, including its own capability probe, but an application talking to the backend port directly bypasses that. Route through the control plane.

Reading logs

Show Log on the row tails the server’s own log, and the panel now survives a status change, so it can be left open across a start. The log is the authoritative account of a failure; the row’s message is a classification of it.

Stopping and restarting

Stop terminates the process and takes the server out of the routing pool; in-flight requests are not waited for. Lifeboat identifies its own processes by a tag it sets on them rather than by matching command lines, so stopping is precise even when several servers run the same model.

A server that crashes is restarted automatically, with a budget. The permanent failures above (gpu_incompatible, insufficient_shared_memory, invalid_extra_args) disable retries — retrying something that cannot succeed only obscures the reason.

Sources and references

Which GPU is holding what: Control plane and dashboard
Routing across several servers: Clusters and load balancing
Registering and downloading weights: Model management
Testing a server once it is up: Playground