Quickstart on vLLM

On vLLM, LifeboatCache runs by default in native mode: it is a secondary tier of vLLM’s own KV offloading (OffloadingConnector with TieringOffloadingSpec). vLLM moves KV between the GPU and a host-RAM tier; LifeboatCache persists those blocks to disk or shared storage and hands them back on a hit. No LMCache is needed, and hybrid Mamba / linear-attention models work.

What do I need first?

vLLM 0.32.x on NVIDIA GPUs (other backends: see Supported versions).
pip install lifeboatcache in the same environment (Install).
Host RAM for vLLM’s CPU tier: engine.native_cpu_gb, 20 GB by default. vLLM’s offloading also allocates roughly 19 GB of its own next to it, so budget about native_cpu_gb + 20 GB.
/dev/shm space: vLLM keeps that CPU tier in one file in /dev/shm. Check it:

df -h /dev/shm        # needs more free space than engine.native_cpu_gb

In a container, /dev/shm is only 64 MB unless you pass --shm-size (or --ipc=host) to docker run. When the tier does not fit, LifeboatCache shrinks it to what fits and logs the fix (see Troubleshooting).

How do I start vLLM with LifeboatCache?

LIFEBOAT_CACHE=1 vllm serve Qwen/Qwen3-30B-A3B \
  --served-model-name my-model \
  --enable-prefix-caching \
  --kv-cache-dtype fp8

That is the whole change. --kv-cache-dtype fp8 is optional; with it, stored KV is already 8-bit and is written raw to local disk.

The same wiring, written out explicitly (no environment variable; useful when you manage the --kv-transfer-config yourself):

vllm serve Qwen/Qwen3-30B-A3B --served-model-name my-model --enable-prefix-caching \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":21474836480,
      "secondary_tiers":[{"type":"LifeboatTier","module_path":"lifeboat_cache.integrations.vllm_tier"}]}}'

cpu_bytes_to_use is the CPU tier in bytes (21474836480 = 20 GiB). To point the tier at a specific configuration file, add "lifeboat_config": "/etc/lifeboat.yaml" to the LifeboatTier entry.

How do I know it attached?

Look for these lines in vLLM’s log at startup:

LIFEBOAT_CACHE=1: Lifeboat Cache via vLLM native offloading
Lifeboat vLLM tier: namespace=vllm-3f9c… rank=0 block=… bytes tiers=['local-fallback']
Lifeboat Cache dashboard: http://127.0.0.1:9400/

With no configuration file you will also see no persistent tier available (none configured); writing KV to local fallback … — expected: KV goes to ~/.lifeboat_cache/kv. Configure a real tier in Tiers & storage backends.

On a model whose layers all use one KV block size, LifeboatCache offloads 256-token blocks (vLLM’s own default is its 16-token GPU block, which makes one small file per block). On models that mix sliding-window and full attention or Mamba layers it leaves vLLM’s per-group block sizes alone and logs native_block_tokens=256 not used: … has KV-cache groups with different block sizes.

How do I prove a prompt is served from the cache?

Send a long prompt (several thousand tokens) and note the time to first token.
Restart vLLM (or evict it from the GPU by sending other long prompts).
Send the same prompt again and compare.

python - <<'EOF'
import json, time, urllib.request
doc = " ".join(f"Section {i}: the quarterly report lists revenue, costs and risks." for i in range(900))
body = {"model": "my-model", "prompt": doc + "\nSummarize section 42.", "max_tokens": 1, "stream": False}
req = urllib.request.Request("http://localhost:8000/v1/completions",
                             data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
t = time.time(); urllib.request.urlopen(req).read(); print(f"{time.time() - t:.2f} s")
EOF

Then read the cache’s own counters:

lifeboat-cache status
# engine vllm-native  up …s  policy lru
# hit rate      …  (… lookups, … loads served)
# index         … chunks, … MiB
#  * local-fallback   filesystem  closed …

After the restart the log also reports recovered N entries (… MiB) from disk in …s: the index is rebuilt in the background from the tier’s manifest.

Which settings matter on vLLM?

Key (`lifeboat.yaml`)	Default	When to change it
`engine.native_cpu_gb`	`20`	Raise it when several long prompts return at once: everything restored from disk passes through this tier
`engine.native_cpu_fit_shm`	`true`	`false` fails at startup with the fix instead of shrinking the tier to fit `/dev/shm`
`engine.native_block_tokens`	auto (256 where valid)	`0` keeps vLLM’s default block size
`engine.native_promotion_wait_s`	`5.0`	How long a request whose blocks are on disk waits for room in the CPU tier before vLLM recomputes it; `0` turns the wait off
`storage.fallback_local.capacity_gb`	`50`	The disk budget when no tier is configured

Any key can also be set from the environment, e.g. LIFEBOAT__engine__native_cpu_gb=40. Full list: Configuration reference.

For hybrid (Mamba / linear-attention) models, keep --max-num-seqs below the model’s Mamba cache-block count.

What if I already use LMCache?

An explicit --kv-transfer-config always wins — LIFEBOAT_CACHE=1 never replaces a connector you configured. To put LifeboatCache under your own LMCache server instead, see vLLM native vs LMCache mode.