Skip to content
Docs

FAQ

Is LifeboatCache the same thing as Lifeboat?

Section titled “Is LifeboatCache the same thing as Lifeboat?”

No. Lifeboat is Iterate.ai’s inference platform. LifeboatCache is a separate Python package — a tiered, persistent KV cache — that runs on top of vLLM, SGLang or Lifeboat. You can use LifeboatCache without Lifeboat, and Lifeboat without LifeboatCache.

Do I have to change my engine or fork it?

Section titled “Do I have to change my engine or fork it?”

No. pip install lifeboatcache registers plugins through the engines’ own extension points, and LIFEBOAT_CACHE=1 turns them on. Without the variable, or with an explicit --kv-transfer-config (vLLM) or --hicache-storage-backend (SGLang), the launch is unchanged.

Does it change model output?

Section titled “Does it change model output?”

Not with the defaults on vLLM native mode, which stores blocks raw, or with the lossless codec. With the default fp8 codec on SGLang, Lifeboat or LMCache mode, restored KV from a bf16 cache is 8-bit, the same trade-off as running the engine with an fp8 KV cache; an fp8 KV cache is stored without further loss. The LBQ low-bit codecs are opt-in and lossy to a degree that depends on bit width — lbq2 is visibly lossy. Model weights are never touched. Note that vLLM with an fp8 KV cache is not bit-deterministic even without a cache, so compare outputs on quality, not bytes.

How much host RAM and disk does it need?

Section titled “How much host RAM and disk does it need?”
  • Host RAM: the engine’s host-RAM tier. vLLM native mode: engine.native_cpu_gb (20 GB) plus roughly 20 GB vLLM allocates next to it. SGLang: the HiCache host pool (2–16 GB per rank by default). LMCache mode: LMCache’s L1, auto = 40% of available RAM.
  • Disk: whatever you budget with capacity_gb — 50 GB for the default local tier — and LifeboatCache always leaves min_free_gb (10 GB) free.

More host RAM lets more returning prompts be staged at once; more disk lets more distinct prompts be kept. Less DRAM, less performance; more DRAM, better performance.

What happens when the server restarts?

Section titled “What happens when the server restarts?”

The KV on disk stays. At startup each store rebuilds its index from the tier’s manifest in the background, and prompts stored before the restart are restored on their next visit instead of recomputed. Only KV still being written at the moment of shutdown is lost, and it is simply recomputed.

What happens when storage fails?

Section titled “What happens when storage fails?”

A tier that fails its startup probe is skipped and the next tier (ultimately the local fallback) takes the writes. At runtime a failing tier’s circuit breaker opens and it is skipped for 30 seconds. A read that fails, or a block that fails its integrity check, is treated as a miss and recomputed. Storage problems cost speed, never correctness, and never stop the engine.

Does it help if my prompts never repeat?

Section titled “Does it help if my prompts never repeat?”

No. A cache with nothing to reuse is overhead: by default every page is still written, which costs storage bandwidth. On SGLang or Lifeboat with mostly-unique traffic, set LIFEBOAT_SGLANG_WRITE_POLICY=write_through_selective so only pages that are hit again are written; on any engine, measure with and without LIFEBOAT_CACHE=1 before deploying.

How is it different from vLLM’s own disk tier or LMCache’s?

Section titled “How is it different from vLLM’s own disk tier or LMCache’s?”

Both engines ship offload tiers; persistence itself is where most of the gain over GPU-only serving comes from. LifeboatCache adds: storage targets with probing and fallback (local, NFS, ONTAP, S3), per-tier compression, an enforced whole-server capacity budget with a configurable eviction policy, integrity checking, indexed recovery, sharing one directory across engine processes, waiting for host-tier room instead of recomputing returning users (vLLM native), and a dashboard with Prometheus metrics. See Benchmarks for a measured comparison against vLLM alone and vLLM with LMCache.

Is the dashboard safe to expose?

Section titled “Is the dashboard safe to expose?”

It binds to localhost by default. Expose it only with dashboard.auth_token set; configuration writes also require a JSON content type and an allowed Host. Set dashboard.allow_config_edit: false for a read-only view.

What are the current limitations?

Section titled “What are the current limitations?”
  • AMD and Intel GPUs are expected to work through the engines’ offloading but have not been measured with LifeboatCache.
  • SGLang on Apple silicon: hybrid models one request at a time (an SGLang MLX runtime crash).
  • vLLM LMCache mode: standard-attention models only; a single request larger than about 80% of LMCache’s L1 cannot be staged. Native mode has neither limit.
  • Large SGLang host pools (LIFEBOAT_SGLANG_HOST_GB=auto or tens of GB per rank) are opt-in: validate GPU memory headroom and throughput under your peak load first.
  • NIXL RDMA and DOCA tier types are not validated on hardware.
  • Cross-node sharing through one NFS or ONTAP directory works for lookups, but NFS tiers enforce capacity per process.

Where do I get help?

Section titled “Where do I get help?”

Troubleshooting lists every log line with its cause and fix. For anything else, contact Iterate.ai through iterate.ai/contact.

Sources and references

Section titled “Sources and references”