Compression
LifeboatCache can store KV smaller than the engine holds it. Each storage tier gets its own codec through the compression ladder, so fast local disks can favour restore speed while shared and archive storage favour bytes.
Which integrations compress KV?
Section titled “Which integrations compress KV?”| Integration | Stored as |
|---|---|
| SGLang and Lifeboat (HiCache backend) | the tier’s ladder codec |
| vLLM in LMCache mode | the tier’s ladder codec |
| vLLM in native mode (the default) | raw bytes — its blocks mix attention KV and recurrent state, which must not be quantized |
| Hybrid-model recurrent (Mamba / linear-attention) state, any engine | raw, lossless |
On vLLM native mode, run the engine with --kv-cache-dtype fp8 if you want 8-bit KV on disk:
the engine’s own KV is then already half the size of bf16.
Which codec does each tier use?
Section titled “Which codec does each tier use?”The tier’s type picks its rung of the ladder:
| Rung | Tier types | Default codec |
|---|---|---|
cpu |
memory |
fp8 |
local_nvme |
filesystem, nvme |
fp8 |
shared |
nfs, ontap |
fp8 |
archive |
s3 |
fp8 |
compression: ladder: {local_nvme: fp8, shared: lbq4, archive: lbq3} lossless_8bit: remote # off | remote | on"none" on a rung stores raw bytes. A codec named in the ladder must exist in
compression.codecs (the built-in set is always available); otherwise validation fails with
compression.ladder.<rung> references codec '<name>' not defined in compression.codecs.
What codecs are available?
Section titled “What codecs are available?”| Codec | Kind | Size vs bf16 KV | Use it for |
|---|---|---|---|
fp8 (default) |
8-bit float with scale | 2× smaller | everywhere restore speed matters — it decodes fastest |
lossless (zkv) |
exact: byte planes + zstd | about 1.46× on bf16, about 1.2× on fp8 KV | exact storage; max_accuracy preset |
lbq4 (alias turboquant4) |
LBQ 4-bit keys / 4-bit values | about 3.5–3.8× | shared and archive storage, capacity-bound disks |
lbq3 (alias turboquant3) |
LBQ 4-bit keys / 2-bit values | about 4.3–5.0× | the storage_efficient preset |
lbq2 (alias oscar2) |
LBQ 2-bit / 2-bit, first and last layer at fp8 | about 5–6× | lossy; only where capacity dominates |
kvtc |
PCA + quantized coefficients | about 1.9× | near-lossless, slow |
LBQ is LifeboatCache’s low-bit codec. For each stored chunk it centres the keys (attention ignores a shared key offset), applies a randomized Hadamard rotation, stores a per-vector scale and codes values with a Lloyd-Max codebook, with separate key and value bit widths. The first and last layers and the chunk that opens a sequence (the attention sink) stay at fp8 where the engine reports chunk positions (LMCache mode); SGLang pages carry no position, so no sink protection applies there.
The low-bit codecs decode several times slower than fp8 on a CPU, so the defaults keep fp8 on every rung; use LBQ where bytes cost more than CPU time. Size ratios above were measured on a 4B-parameter model’s bf16 KV and vary by model.
What happens with fp8 KV caches?
Section titled “What happens with fp8 KV caches?”An engine running --kv-cache-dtype fp8 hands over KV that is already 8-bit, which the lossy
codecs cannot shrink further. compression.lossless_8bit decides what happens:
| Value | Local rungs | Shared / archive rungs |
|---|---|---|
remote (default) |
stored raw | lossless (zkv) — those links are slower than the decode |
on |
lossless | lossless |
off |
raw | raw |
On a fast local NVMe, decoding costs more time than the roughly 20% of bytes it saves, which is
why remote is the default.
Where are blocks decoded?
Section titled “Where are blocks decoded?”compression.decode_device chooses where fp8 and LBQ blobs are decoded on restore: auto
(default) uses the CUDA / ROCm GPU when one is visible and falls back to the CPU on device
errors; cpu forces the CPU codecs; mps (Apple) is opt-in. On SGLang and Lifeboat, auto
decodes on the CPU, because restores land in the CPU host pool — only an explicit cuda
decodes there on the GPU. Lossless always decodes on the CPU, in parallel threads.
How is accuracy monitored?
Section titled “How is accuracy monitored?”With the accuracy guard enabled (the default), LifeboatCache samples a share of stored
chunks (sample_rate, 1% by default), decodes them again and measures the reconstruction error
of the live codec. The dashboard shows the live codec, the compression ratio, bytes saved and
that sampled error, so a lossy codec’s effect is visible rather than assumed. Set
sample_rate: 0 to skip the sampling work.
compression: accuracy_guard: {enabled: true, sample_rate: 0.01}``` The presets `max_accuracy` (lossless on local rungs, a stricter guard) and`storage_efficient` (`lbq3` everywhere) are starting points: `extends: max_accuracy`.
## Sources and references
- [Tiers & storage backends](/lifeboatcache/capabilities/tiers-storage/)- [Configuration reference](/lifeboatcache/platform/configuration-reference/) — `compression` section- [Dashboard & telemetry](/lifeboatcache/capabilities/dashboard-telemetry/)