Model Management
This page is the summary. Each section below has a page of its own:
The registry is the list of models this deployment knows about. Registering a model does not download it and does not start anything — it records where the weights are or will come from.
Register a model
Section titled “Register a model”Models → Add Model, then one of:
- Hugging Face repository — a repo id such as
Qwen/Qwen3-8B. Nothing is fetched until you download it. - Local path — a directory of weights, or a single
.gguffile, already on the host. This is the standard air-gapped workflow. - Recommended — a curated list, filtered to what this machine can actually serve and ordered by what runs best on it. On a Mac that means MLX first, then GGUF; on a GPU server, full-precision checkpoints. A model absent from this list is not unavailable — it is not servable here.
Alias is worth setting. It is the name the API advertises and the name applications use, and it is what stops the raw filesystem path appearing in /v1/models. Update it later with the Models page’s edit action.
Download weights
Section titled “Download weights”Download on the row. Progress reflects real byte and file totals, and a download can be cancelled mid-flight — the registry entry resets immediately so a corrected download can be queued straight away.
The variant picker
Section titled “The variant picker”A Hugging Face repository often holds many quantizations of the same model — sometimes twenty or more, hundreds of gigabytes in total. Lifeboat lists what is actually in the repo, with real per-file sizes, and you tick the one you want.
The Download button stays disabled until the listing has loaded and you have chosen something. That is deliberate: without it, clicking early pulls the entire repository, which is where the 100 GB surprises came from.
If the listing cannot be read you get a warning panel and the button is re-enabled, so you can still proceed with a full-repository download — with the consequence stated rather than hidden.
“This model cannot be served on this machine”
Section titled ““This model cannot be served on this machine””The picker refuses the download and explains why, before you click, when the repository holds only full-precision weights and this host has no tensor engine — every Mac, and any Linux machine without CUDA or ROCm. A 60 GB download that could never load is worth preventing.
What to do: choose a GGUF build of the same model. A repository holding both formats is not blocked — its GGUF variants are still offered.
If a download does fail with format_not_servable, that is the same check running server-side on a path the picker could not judge.
Other download failures
Section titled “Other download failures”| Message | Cause |
|---|---|
model_download_failed |
Network, or the repository is gated. Add a Hugging Face token — see Integrations. |
model_too_large |
Not enough disk in the models directory. |
remote_disk_full |
Same, on a remote node during replication. |
model_not_present_on_node |
A remote node was asked to serve weights it does not have. Replicate first. |
Downloads to several nodes at once are explicit, not automatic: choose the nodes when starting the download. Nothing is synchronized between nodes in the background.
Which format do I want?
Section titled “Which format do I want?”| Format | Where it runs | Notes |
|---|---|---|
| Safetensors (full precision, FP8, FP4) | NVIDIA and AMD GPUs, via the tensor engine | The fast path. Gets the full optimization layer and speculative decoding. |
| GGUF (quantized) | Anywhere, including CPU-only and integrated GPUs | Much smaller. Runs on the GGUF engine, which does not get the optimization layer. |
| MLX | Apple Silicon only | The fast path on a Mac. |
Lifeboat picks the engine for you. An explicit choice always wins, but the default routes a GGUF the tensor engine cannot handle to the GGUF engine, and re-checks at start when the weights have arrived since the server was created.
On a 4-bit quantization: it reads half the bytes per token of an 8-bit one, and decode speed is bound by memory bandwidth, so on a memory-constrained machine the quantized build is usually faster as well as smaller.
Capability probing
Section titled “Capability probing”Probe asks a running server what the model actually supports — streaming, real maximum output length, effective context window, JSON mode, tool calling, vision, reasoning traces — instead of trusting the model card.
The result is cached and reused. If a probe looks wrong after an upgrade, Refresh probe re-runs it; Lifeboat also re-probes automatically when a stored result was produced by a version known to have recorded it incorrectly.
One number to treat carefully: the context window reported here is the limit a client must stay under, which is the smaller of the model’s window and what the cache can hold. It is deliberately not the cache size, which is shared across concurrent requests and is much larger.
Automatic draft models
Section titled “Automatic draft models”When a model finishes downloading and Lifeboat recognizes it, the matching speculative-decoding draft head is fetched alongside it and recorded on the row. Nothing is required from you, and a failed fetch leaves the model fully usable without speculative decoding. See Servers.
Delete a model
Section titled “Delete a model”Two different actions:
- Remove from registry (default) — forgets the model. The weights stay on disk.
- Delete files from disk — removes the weights too. Restricted to the approved models directories, and the registry row is only removed after the disk operation succeeds, so a partial failure leaves something to retry rather than an orphan.
For a single .gguf file, the parent directory is cleaned up if it becomes empty — so deleting one quantization out of a multi-quant repository leaves its siblings alone.
Replicas on other nodes are not swept. Delete those on each node.
Sources and references
Section titled “Sources and references”- Serving a registered model: Servers and multi-GPU
- Gated repositories and tokens: Integrations
- What the API advertises: API and automation