Skip to content
Docs

GGUF Variants

A single GGUF repository commonly holds twenty or more quantizations of the same model. Downloading all of them can be hundreds of gigabytes; you want one.

The picker

Section titled “The picker”

Open Download on a GGUF model and Lifeboat lists what is actually in the repository — each variant with its real size — as a checklist. Tick one (or several) and download only those.

The Download button starts disabled and enables only once (a) the listing has resolved and (b) at least one variant is ticked. That gating is deliberate: without it, clicking before the listing arrives pulls the entire repository. That was the original defect, and it cost people 100 GB+ pulls from multi-quant repositories.

Reading quantization names

Section titled “Reading quantization names”

Names like Q4_K_M, Q5_K_S, Q8_0 encode bits per weight and the strategy. What matters operationally:

  • Fewer bits = smaller and faster. Decode is memory-bandwidth bound, so a 4-bit build reads half the bytes per token of an 8-bit one and is usually about twice as fast, as well as half the size.
  • Fewer bits = more quality loss, gradually. 4-bit is widely considered the sweet spot for most uses; below that, degradation becomes noticeable.
  • On a memory-constrained machine the quantized build is the better choice on both axes, not a compromise.

Sharded sets

Section titled “Sharded sets”

Some repositories split one quantization across several files. The tensor engine requires a single file and Lifeboat preflights this, surfacing a clear error rather than letting it fail at server start — merge it first with the GGUF tooling’s merge command.

The GGUF engine accepts a sharded set via its first shard, so a sharded model is servable there.

Vision models

Section titled “Vision models”

A GGUF vision model needs a separate projector file alongside the weights. Lifeboat finds it automatically, searching the weights directory and one level up — repositories commonly keep the projector at the root while the weights sit in a per-quantization subdirectory. You can override the choice in the server’s Advanced settings.

When the listing cannot be read

Section titled “When the listing cannot be read”

You get a warning panel and the button is re-enabled, so you can still proceed with a full-repository download — with the consequence stated rather than hidden. That is the deliberate trade: refusing outright would block a legitimate download over a listing failure.

Sources and references

Section titled “Sources and references”