Supported Versions
Current release: LifeboatCache 0.1.23 (pip install lifeboatcache==0.1.23).
Which engine versions are supported?
Section titled “Which engine versions are supported?”| Component | Supported range | Measured with |
|---|---|---|
| Python | 3.10 or newer | 3.12 |
| vLLM | >=0.32.0,<0.33 |
0.32.0 |
| SGLang | >=0.5.19,<0.6 |
0.5.19 (NVIDIA), 0.5.22 (Apple silicon, MLX) |
| LMCache (vLLM LMCache mode only) | >=0.5.6,<0.6 |
0.5.6 |
| Lifeboat | container images with the hierarchical KV cache | 2.2.61 |
boto3 (S3 tiers only) |
1.28 or newer | — |
LifeboatCache attaches through each engine’s own extension points — vLLM’s general plugins and offloading tiers, SGLang’s plugins and HiCache storage interface — so it does not patch or fork the engine. Those interfaces are what the version ranges pin.
Which accelerators and platforms does it run on?
Section titled “Which accelerators and platforms does it run on?”| Engine and backend | Status |
|---|---|
| vLLM on NVIDIA GPUs (CUDA) | Supported and measured (RTX PRO 6000 Blackwell) |
| vLLM on AMD (ROCm) and Intel (XPU) GPUs | vLLM’s offloading supports them; not yet measured with LifeboatCache |
| vLLM on CPU (Apple silicon, x86 and ARM servers) | Supported in native mode; LMCache mode needs a GPU |
| SGLang on NVIDIA GPUs (HiCache) | Supported and measured |
| SGLang on AMD GPUs (HiCache) | Expected to work through HiCache; not yet measured |
| SGLang on Apple silicon (MLX backend) | Supported as SGLang’s prefix-cache backend; hybrid models one request at a time |
| Lifeboat containers on NVIDIA GPUs | Supported and measured |
MLX (mlx-lm), llama.cpp, Ollama |
Not integrated |
Operating systems: Linux for production; macOS for development and Apple-silicon serving.
O_DIRECT reads and the nvme tier type are Linux features; elsewhere they fall back to
ordinary buffered file I/O.
What happens on a platform it cannot attach to?
Section titled “What happens on a platform it cannot attach to?”LIFEBOAT_CACHE=1 leaves the launch exactly as it would be without it and logs why — for
example on vLLM:
LIFEBOAT_CACHE=1: Lifeboat Cache in lmcache mode isn't available on vLLM's cpu backend; it stays offThe engine starts normally. LifeboatCache never blocks an engine from starting; the only
deliberate startup failure is engine.native_cpu_fit_shm: false when vLLM’s CPU tier cannot fit
/dev/shm, which you opt into.
Which model types are supported?
Section titled “Which model types are supported?”| Model type | vLLM native | vLLM LMCache mode | SGLang / Lifeboat |
|---|---|---|---|
| Standard attention (dense and MoE) | yes | yes | yes |
| MLA (KV replicated across TP ranks) | as vLLM’s offloading supports it | as LMCache supports it | yes; ranks share one shard |
| Sliding-window + full attention (e.g. gpt-oss, Gemma) | yes; per-group block sizes kept | — | yes |
| Hybrid Mamba / linear attention (e.g. Qwen3.5 / 3.8 hybrids) | yes, KV and recurrent state | limited — LMCache’s L1 cannot stage long hybrid prompts; use native mode | yes, state stored losslessly |
Is there anything upstream I should know about?
Section titled “Is there anything upstream I should know about?”- vLLM 0.32 FlashInfer CUTLASS MoE kernel on RTX PRO 6000 can abort under concurrent load
with
GPU lacks the shared memory resources to run fused_moe kernel. This is in vLLM / FlashInfer, with or without LifeboatCache; start vLLM with--moe-backend triton. - vLLM killed with SIGKILL can leave
/dev/shm/vllm_offload_*.mmapbehind, using/dev/shmspace. Delete it when no vLLM process maps it. - SGLang MLX, hybrid models: concurrent requests crash inside SGLang’s MLX runtime; run them one at a time.