Troubleshooting
Start with two commands: the engine’s log (LifeboatCache logs under lifeboat_cache.* and its
messages start with LIFEBOAT_CACHE=1: or Lifeboat Cache), and
lifeboat-cache statuswhich shows whether a store is running, its hit rate, and every tier with its health.
Is LifeboatCache running at all?
Section titled “Is LifeboatCache running at all?”| Symptom | Cause | Fix |
|---|---|---|
No LIFEBOAT_CACHE=1: … line in the engine log |
the variable is not set in the engine’s environment, or the package is in a different Python environment | LIFEBOAT_CACHE=1 in the same shell or service unit as the engine; python -c "import lifeboat_cache" with the engine’s interpreter |
no Lifeboat node answering at http://127.0.0.1:9400: … Connection refused |
no store running, the dashboard is disabled, or you are on a host without the rank-0 process | check the log line Lifeboat Cache dashboard: …; on SGLang ask the node running TP rank 0 |
vLLM ignores LIFEBOAT_CACHE=1 |
an explicit --kv-transfer-config was given — it always wins |
remove it, or add the LifeboatTier to your own OffloadingConnector config (Quickstart on vLLM) |
SGLang ignores LIFEBOAT_CACHE=1 |
the launch set --hicache-storage-backend (or, on Lifeboat, a server’s Storage backend is set in the console) |
remove that choice so LifeboatCache becomes the backend |
SGLang plugin hooks unavailable; pass the --hicache-* flags explicitly (see lifeboat_cache.integrations.sglang_backend) |
the SGLang build has no plugin hook for Python-API launches | pass the explicit flags from Quickstart on SGLang |
LIFEBOAT_CACHE=1: Lifeboat Cache in lmcache mode isn't available on vLLM's cpu backend; it stays off |
LMCache mode needs a GPU; other vLLM backends have no offloading | use engine.vllm_mode: native on CPU; on unsupported backends the cache stays off by design |
LIFEBOAT_CACHE=1 but the radix cache is disabled and LIFEBOAT_CACHE_FORCE_RADIX=0; HiCache can't run without it, so the Lifeboat backend stays off for this launch |
the launch disabled SGLang’s prefix cache and you asked LifeboatCache to respect that | unset LIFEBOAT_CACHE_FORCE_RADIX (default 1 re-enables the prefix cache) |
could not enable the Lifeboat HiCache backend followed by a traceback |
an unexpected error while filling SGLang’s arguments; SGLang starts without the cache | report the traceback; meanwhile use the explicit --hicache-* flags |
Why does vLLM fail to start, or start with a smaller CPU tier?
Section titled “Why does vLLM fail to start, or start with a smaller CPU tier?”| Symptom | Cause | Fix |
|---|---|---|
Lifeboat Cache: vLLM's CPU KV tier of 20.0 GiB doesn't fit in /dev/shm (… of … GiB free); using N GiB. For the full tier, make /dev/shm larger … |
vLLM keeps its CPU tier in one /dev/shm file; LifeboatCache shrank it to fit |
docker run --shm-size=32g (or --ipc=host), remount the tmpfs larger, delete stale /dev/shm/vllm_offload_*.mmap, or lower engine.native_cpu_gb |
startup fails with … doesn't fit in /dev/shm … (ShmTooSmall) |
engine.native_cpu_fit_shm: false, or less than 1 GiB fits |
as above |
vLLM’s own Insufficient space in /dev/shm |
a CPU tier configured without LifeboatCache’s fit check | as above |
/dev/shm full although vLLM is not running |
a vLLM killed with SIGKILL left /dev/shm/vllm_offload_*.mmap |
delete the file when no process maps it (lsof /dev/shm/vllm_offload_*) |
Lifeboat Cache: native_block_tokens=256 not used: <model> has KV-cache groups with different block sizes … |
sliding-window / full-attention or Mamba layers: vLLM needs a block size per group | informational; nothing to do |
Lifeboat Cache: native_block_tokens=256 not used: can't read <model>'s config.json … |
the model’s config.json is neither local nor downloadable |
informational; pass a local model path to get 256-token blocks |
vLLM fails at startup after you set engine.native_block_tokens on releases before 0.1.21 |
mixed KV-cache groups (e.g. Gemma) | upgrade; current releases skip the setting with the warning above |
Lifeboat Cache: vLLM async scheduling off for <model> (engine.native_async_scheduling) |
a sliding-window + full-attention model (gpt-oss, Gemma): with vLLM 0.32’s async scheduling, its offload copies raced the model under a full CPU tier and the engine stopped with CUDA error: an illegal memory access |
informational; the setting avoids the crash at some throughput cost. engine.native_async_scheduling: true or --async-scheduling turns it back on |
CUDA error: an illegal memory access was encountered with LifeboatCache on gpt-oss / Gemma, before 0.1.24 |
the same race | upgrade to 0.1.24, or start vLLM with --no-async-scheduling |
GPU lacks the shared memory resources to run fused_moe kernel |
a vLLM / FlashInfer MoE kernel issue under concurrency, independent of LifeboatCache | --moe-backend triton |
Why is a storage target not being used?
Section titled “Why is a storage target not being used?”| Log line | Cause | Fix |
|---|---|---|
tier <name> failed its startup probe (…); skipping |
the write / read-back / delete test failed: permissions, read-only mount, full disk | fix the path or mount; touch <path>/x as the engine’s user |
tier <name> unavailable: ValueError: tier '<name>' (type filesystem) requires a 'path' |
missing required key | add path (or bucket for S3, mount/path for ONTAP) |
tier <name> unavailable: OSError: nfs tier '<name>': /mnt/nfs is not a mount point |
the export is not mounted | mount it before starting the engine |
tier <name> unavailable: RuntimeError: s3 tier '<name>' requires boto3 (not installed) |
S3 support not installed | pip install "lifeboatcache[s3]" |
no persistent tier available (…); writing KV to local fallback <path> |
every configured tier was skipped, or none is configured | expected with no config; otherwise fix the tiers listed in the message |
ontap tier '<name>': protocol=pnfs_rdma requested but mount is not pNFS-capable (…); continuing with plain NFS/file I/O (fail-safe) |
the mount is not NFSv4.1 (+RDMA) | remount with pNFS, or drop protocol; strict_protocol: true makes this fatal |
tier <name>: SQLite manifest unavailable (…); using directory scans |
the manifest could not be opened | check the directory’s permissions; recovery still works, more slowly |
a tier shows open on the dashboard; lifeboat_tier_breaker_open{tier=…} 1 |
more than 20% of its operations failed; it is skipped for 30 s, then retried | check the storage; tune storage.backpressure.circuit_breaker |
Why is the hit rate zero or low?
Section titled “Why is the hit rate zero or low?”| Symptom | Cause | Fix |
|---|---|---|
| hit rate 0% after a restart on vLLM | the server’s layout changed: model, TP size, KV dtype or block size are part of the key | restart with the same flags; a different layout is a different cache by design |
| hit rate 0% after a restart on SGLang | the restart came before the background writes finished, or the page size changed | allow a few seconds after long prompts; keep --page-size constant |
| high hit rate on the dashboard, little speed-up | the engine’s host-RAM tier is too small to stage what returns at once | raise engine.native_cpu_gb (vLLM) or LIFEBOAT_SGLANG_HOST_GB (SGLang) |
| hit rate falling while evictions climb | the storage budget is below the working set | raise capacity_gb, add a tier, or use a smaller codec on it |
| prompts never repeat | nothing to reuse — the cache cannot help | expected; see FAQ |
| hybrid model on vLLM LMCache mode restores little | LMCache does not stage long hybrid prompts | use engine.vllm_mode: native |
corrupt chunk … -> treated as miss |
a stored block failed its integrity check (damaged file, partial copy) | nothing to do: it is recomputed and rewritten; check the disk if it recurs (lifeboat_corrupt_total) |
vLLM FileMapper unavailable; namespacing by model name only |
a vLLM build outside the supported range | use vLLM 0.32.x |
Why can’t I reach or edit the dashboard?
Section titled “Why can’t I reach or edit the dashboard?”| Response | Cause | Fix |
|---|---|---|
| connection refused from another machine | the dashboard binds to 127.0.0.1 |
ssh -L 9400:localhost:9400 <node>, or set dashboard.host: 0.0.0.0 with dashboard.auth_token |
401 {"error": "unauthorized"} |
a token is configured and was not sent | open with ?token=<token> or send Authorization: Bearer <token> |
403 {"error": "bad Host header"} on save |
editing through a proxy host name | add the name to dashboard.allowed_hosts |
403 … config editing is disabled (dashboard.allow_config_edit: false) |
read-only dashboard | set allow_config_edit: true |
dashboard port 9400 unavailable (…); another process (e.g. a different TP rank) is probably serving it |
several processes on one host want the same port | expected for TP ranks; for separate servers give each its own dashboard.port |
prometheus port 9402 unavailable (…); /metrics is still on the dashboard port |
port taken | scrape /metrics on the dashboard port, or change telemetry.prometheus.port |