Skip to content
Docs

Downloading Models

Download on a model row fetches the weights. Progress reflects real byte and file totals, and a download can be cancelled mid-flight — the registry entry resets immediately so a corrected download can be queued straight away.

Before it starts: the servability check

Section titled “Before it starts: the servability check”

If the repository holds only full-precision weights and this host has no tensor engine — every Mac, and any Linux machine without CUDA or ROCm — the download is refused in the picker, with the reason and the remedy, before you click.

A 60 GB download that could never load is worth preventing. Choose a GGUF build of the same model instead. A repository holding both formats is not blocked; its GGUF variants are still offered.

If a download does fail with format_not_servable, that is the same check running server-side on a path the picker could not judge in advance.

The variant picker

Section titled “The variant picker”

A repository often holds many quantizations of one model — sometimes twenty, hundreds of gigabytes in total. Lifeboat lists what is actually there with real per-file sizes and you tick what you want. See GGUF variants.

Gated and private repositories

Section titled “Gated and private repositories”

These need a token on file, and the failure looks like a network problem rather than an authorization one. Add one under HuggingFace Tokens, and accept the model’s licence on the hub with the same account — a token without the acceptance still fails. See Hugging Face tokens.

Multi-node replication

Section titled “Multi-node replication”

Explicit, never automatic. Choose the nodes when starting the download and the control plane fans the pull out to each node’s agent and polls per-node progress.

Before any remote launch the control plane asks the node whether the model is present, and that check understands a directory of weights, a single .gguf file, or a directory containing one — so a GGUF-only replica is correctly seen.

Failures

Section titled “Failures”
Message Cause Fix
model_download_failed Network, or a gated/private repository Add a token and accept the licence
model_too_large Not enough disk in the models directory Free space or point at a larger volume
remote_disk_full Same, on a node during replication Free space on that node
model_not_present_on_node A remote server was asked to serve weights that node does not have Replicate first
format_not_servable No format in the repository this host can serve Choose a GGUF build
It pulled far more than expected The variant picker was bypassed Use the picker and tick one quantization

Automatic draft models

Section titled “Automatic draft models”

When a recognized model finishes downloading, the matching speculative-decoding draft model is fetched alongside it and recorded on the row. It is best-effort: a failure leaves the base model fully usable. See Speculative decoding.

Sources and references

Section titled “Sources and references”