Skip to content
Docs

Overview

LifeboatCache is a tiered, compressed, persistent key-value (KV) cache for LLM inference servers. It runs on top of vLLM, SGLang or Lifeboat — Iterate.ai’s inference platform — without forking the engine: pip install lifeboatcache registers engine plugins, and LIFEBOAT_CACHE=1 turns it on.

When a prompt’s KV no longer fits in GPU memory, the engine evicts it. LifeboatCache keeps those blocks — in host RAM through the engine’s own offload tier, then on local NVMe, NFS, NetApp ONTAP or S3 — so a returning prompt is re-loaded instead of re-computed, including after a full server restart.

Less DRAM, less performance; more DRAM, better performance — achieved just with LifeboatCache on top of vLLM, SGLang or Lifeboat.

What problem does LifeboatCache solve?

Section titled “What problem does LifeboatCache solve?”

Prefill — computing the KV for a prompt — is the expensive part of serving long prompts, and the engine’s prefix cache can only skip it while the KV is still in GPU memory. On a busy server the GPU holds a small fraction of what users send: agent sessions that grow turn by turn, long system prompts and tool manifests, retrieved documents, users who come back to a conversation an hour later. Every one of them that has been evicted is recomputed from scratch.

LifeboatCache extends the prefix cache below the GPU:

Tier What holds it Survives a restart
GPU memory the engine’s own KV cache no
Host RAM the engine’s offload tier (vLLM OffloadingConnector CPU tier, SGLang HiCache host pool, or LMCache’s L1) no
Local disk / NVMe LifeboatCache filesystem tier, or the automatic local fallback yes
Shared storage LifeboatCache nfs, ontap or s3 tiers yes

How does it attach to each engine?

Section titled “How does it attach to each engine?”
Engine Integration Turn it on
vLLM 0.32 (default, native mode) a secondary tier of vLLM’s own OffloadingConnector + TieringOffloadingSpec; no LMCache needed; hybrid Mamba / linear-attention models supported LIFEBOAT_CACHE=1 vllm serve <model>
vLLM 0.32 (LMCache mode) an L2 adapter under LMCache’s multiprocess connector, with an auto-started local LMCache server engine.vllm_mode: lmcache plus LIFEBOAT_CACHE=1
SGLang a HiCache storage backend (--hicache-storage-backend dynamic) LIFEBOAT_CACHE=1 python -m sglang.launch_server ...
Lifeboat the storage tier below Lifeboat’s hierarchical KV cache LIFEBOAT_CACHE=1 in the Lifeboat container, with the package mounted

An explicit engine configuration always wins: a --kv-transfer-config on vLLM or a --hicache-storage-backend on SGLang means LifeboatCache leaves the launch alone. Without LIFEBOAT_CACHE=1 the plugins do nothing. See vLLM native vs LMCache mode and SGLang HiCache backend.

What does it add beyond the engine’s own offloading?

Section titled “What does it add beyond the engine’s own offloading?”
  • Persistence with recovery. Every stored block carries a self-describing header and an integrity hash; the index is rebuilt from a per-tier SQLite manifest in the background at startup, so KV written before a restart is served after it. A block that fails its hash is treated as a miss and recomputed — a damaged file costs time, never a wrong answer.
  • Storage targets with fallback. Tiers are write-probed at startup; an unreachable ONTAP or NFS export or an S3 bucket without credentials is skipped with the reason shown, and KV goes to local disk instead of the cache silently becoming memory-only.
  • Compression per tier. fp8 by default, a lossless codec for fp8 KV on shared tiers, and the low-bit LBQ codecs where bytes cost more than CPU. See Compression.
  • Capacity that is enforced. capacity_gb is a budget, with a free-disk reserve and a configurable eviction policy (LRU, ARC, or a recency-plus-reuse score). See Eviction & capacity.
  • A dashboard and Prometheus metrics on every node: hit rate, bytes served, compression ratio, per-tier health, and a Configure tab that validates and applies changes live. See Dashboard & telemetry.

When does it pay off — and when does it not?

Section titled “When does it pay off — and when does it not?”

It pays off when the same long prefixes come back after the GPU has evicted them: agentic workloads, multi-turn chat with long histories, many users over a shared document set, and any server that is restarted while users are mid-session. The more distinct long prompts you serve, the more host RAM and disk it can put to work — size the tiers above your working set (distinct long prompts × their length).

It does not help traffic that never repeats a prefix: a cache with nothing to reuse is pure overhead, the same rule Lifeboat’s own KV Cache Tiering page states. Very short prompts are also cheaper to recompute than to fetch. See FAQ and the measured results on Benchmarks.

How do I get started?

Section titled “How do I get started?”
  1. Install — pip install lifeboatcache into the engine’s Python environment.
  2. Pick your engine: vLLM, SGLang or Lifeboat.
  3. Open the dashboard on http://127.0.0.1:9400 and watch the hit rate on the second visit of a long prompt.

Sources and references

Section titled “Sources and references”