Skip to content
Docs

Model Formats

Three formats, and which one you want is a property of your hardware rather than a preference.

Format Runs on Engine Notes
Safetensors (full precision, FP8, FP4) NVIDIA and AMD GPUs Tensor The fast path. Gets the optimization layer and speculative decoding.
GGUF (quantized) Anywhere — CPU, integrated GPUs, discrete GPUs GGUF Much smaller. No optimization layer.
MLX Apple Silicon only MLX The fast path on a Mac.

The catalogue is filtered to your machine

Section titled “The catalogue is filtered to your machine”

The Recommended list shows only formats this host can serve, in preference order — MLX first on Apple Silicon, then GGUF; full-precision checkpoints on a GPU server. A model missing from the list is not unavailable, it is not servable here; register it by id or path if you want it anyway.

Refused before the download, not after

Section titled “Refused before the download, not after”

If a repository holds only safetensors and this host has no tensor engine, the picker explains why and disables Download. The alternative — a 60 GB download that fails at server start — is the thing this prevents.

A repository holding both formats is unaffected; its GGUF variants are still offered.

Choosing on a memory-constrained machine

Section titled “Choosing on a memory-constrained machine”

Decode speed is bound by memory bandwidth, so bytes-per-token sets the ceiling. A 4-bit quantization reads half the bytes of an 8-bit one, so on a laptop or a CPU-only server the quantized build is both smaller and faster. There is no trade to agonize over.

On an 8 GB machine: 1.5B–4B at 4-bit is comfortable, 8B is the ceiling, 14B will not fit.

Choosing on a GPU server

Section titled “Choosing on a GPU server”

Prefer safetensors — that is the only path that gets the optimization layer and speculative decoding.

One vendor-specific caveat: on older AMD generations (CDNA2) the 8-bit path is emulated in software and is very slow, so a quantized GGUF is the better choice there despite giving up the optimization layer. From CDNA3 onward the full set applies. See GPU vendors.

Sources and references

Section titled “Sources and references”