Release Notes

Install or upgrade with pip install -U lifeboatcache. Stored KV written by an older release stays readable after an upgrade.

What changed in 0.1.25 to 0.1.28?

These releases make vLLM + LMCache + LifeboatCache (vllm_mode: lmcache) run on AMD GPUs with the LMCache 0.5.3 that vLLM’s ROCm image ships.

0.1.28: LifeboatCache as LMCache’s storage tier decodes restored pages on the CPU when compression.decode_device is auto, since LMCache restores into its own host buffer. On ROCm the GPU decode’s copy back failed on every restore. cuda still decodes on the GPU.
0.1.27: in LMCache mode, LifeboatCache checks at startup that the installed LMCache has GPU kernels for the GPU it runs on and stops with the rebuild command if not. The LMCache in vLLM’s ROCm image is built for MI300-series GPUs only; on an MI210 its transfers crash and vLLM keeps serving with no cache and no error.
0.1.26: starts LMCache’s server process when the installed LMCache does not start it itself.
0.1.25: the LMCache adapter loads under LMCache 0.5.3.

vLLM native mode turns async scheduling off on sliding-window models (such as gpt-oss), where vLLM 0.32’s offload copies could race the model and crash it under a full CPU tier. engine.native_async_scheduling or vLLM’s --(no-)async-scheduling overrides it.
kill -USR2 <engine pid> writes every thread’s Python stack to the server log, for diagnosing a hung engine.

vLLM native mode offloads 256-token blocks by default on models whose KV-cache groups all use one block size. vLLM’s own default, its 16-token GPU block, made one small file, hash and index entry per block and stalled vLLM under many concurrent agent sessions. 0 restores vLLM’s default; models with mixed sliding-window / full-attention or Mamba layers keep vLLM’s per-group block sizes. See Benchmarks.

SGLang / Lifeboat: restored pages decode on the CPU when compression.decode_device is auto, since they land in the CPU host pool; decoding on the GPU used memory outside SGLang’s budget. An explicit cuda still decodes on the GPU.
vLLM: for a model not yet downloaded, only its config.json is fetched to check its KV-cache groups; if it cannot be read, native_block_tokens is skipped with a warning instead of risking a failed start.

capacity_gb is a whole-server budget on SGLang and Lifeboat. Each tensor- or pipeline-parallel rank now enforces 1/(TP × PP) of it; before, every rank enforced the full amount. Upgrade if you run SGLang or Lifeboat with tensor parallelism.
vLLM no longer fails to start when engine.native_block_tokens is set on a model whose layers form KV-cache groups with different block sizes (sliding-window + full attention, Mamba hybrids): the setting is skipped with a warning.
SGLang’s GPU write-back staging buffer is bounded to 1,024 tokens (LIFEBOAT_SGLANG_STAGING_TOKENS); with 64-token pages it had grown 64-fold.
New, opt-in: LIFEBOAT_SGLANG_HOST_GB=auto sizes the HiCache host pool from GPU and host memory; LIFEBOAT_SGLANG_WRITE_POLICY sets the HiCache write policy.

vLLM’s CPU KV tier is sized to fit /dev/shm: when it does not fit (a container without --shm-size has 64 MB), the tier is shrunk to what fits with a warning naming the fix, instead of vLLM failing to start. engine.native_cpu_fit_shm: false fails with the fix instead.

SGLang zero-copy backups (batch_set_v1 / batch_get_v1): pages are encoded from views of the host pool on LifeboatCache’s workers instead of being copied out on SGLang’s backup thread, so long sessions persist several times faster. LIFEBOAT_SGLANG_ZERO_COPY=0 opts out; the stored bytes are identical.
LMCache mode: LMCache is started with one prefetch in flight (engine.lmcache_prefetch_max_in_flight), so concurrent prefetches of long prompts no longer crowd each other out of the L1.

0.1.18: SGLang on Apple silicon writes finished chunks without holding Python’s global lock, so writes no longer slow the engine.
0.1.17: hybrid Mamba / linear-attention models on SGLang’s MLX backend (Apple silicon).
0.1.16:
- Lifeboat: hybrid models whose prefix cache Lifeboat disables are served with the prefix cache re-enabled for HiCache (LIFEBOAT_CACHE_FORCE_RADIX=0 opts out) and the lazy Mamba buffer, which keeps more concurrent requests; pages are written through in parallel, so long sessions survive a restart.
- Restores decode fp8 and LBQ on the GPU (compression.decode_device).
- Faster native vLLM tier writes; repeat reads served from the page cache after a first O_DIRECT read.
- Shared tier directories: capacity_gb covers the whole directory with one evicting process.
- lifeboat-controld --serve fleet controller; engine.lmcache_l1_size_gb: auto.

0.1.15: SGLang on Apple silicon prefetches returning users’ chunks from request arrival (LIFEBOAT_MLX_PREFETCH_GB).
0.1.14: several returning users at once on SGLang MLX; writes moved off the scheduler.
0.1.13: vLLM on CPU (Apple silicon, x86 and ARM servers) in native mode, and SGLang on Apple silicon as its prefix-cache backend.
0.1.12: LIFEBOAT_CACHE=1 never breaks a launch on a platform where LifeboatCache cannot attach; it logs why and stays off.
0.1.11: storage.shared_lookup for several engine processes sharing one tier directory.

0.1.10: opt-in prefix-cache override for hybrid models on SGLang (made the default in 0.1.16).
0.1.9: the SGLang plugin steps aside instead of failing a launch whose prefix cache is disabled.
0.1.8: promotion wait in vLLM native mode — a returning request whose blocks are on disk waits up to engine.native_promotion_wait_s for CPU-tier room instead of being recomputed.
0.1.7: asynchronous lookups in the vLLM tier, through vLLM’s own lookup thread.
0.1.6: O_DIRECT cold reads, one copy per stored chunk, faster restores into LMCache, and the exact lossless codec; fp8 KV caches are coded losslessly on shared / archive tiers and stored raw on local ones (compression.lossless_8bit).
0.1.5: the LBQ low-bit codec replaces the earlier 3-bit codec; turboquant4/3 and oscar2 now name LBQ rungs; parallel native restores; score-based eviction (recency plus reuse, tail-first) replaces topic-first eviction as the episodic behaviour.

0.1.4: hybrid-model restores under LMCache mode fixed.
0.1.3: write-queue memory leak fixed (upgrade from 0.1.2); LMCache L2 writes bounded (engine.l2_inflight_gb); min_free_gb free-disk reserve; per-tier SQLite manifest for fast restart recovery; engine.native_block_tokens.
0.1.2: vLLM native mode — LifeboatCache as a secondary tier of vLLM’s own offloading, no LMCache needed, hybrid models supported; engine.vllm_mode.
0.1.1: hybrid models on SGLang (recurrent state persisted losslessly), zero-copy restores, dashboard.allowed_hosts, the configuration file saved with owner-only permissions.
0.1.0: first installable release — pip install lifeboatcache, engine plugins for vLLM and SGLang, persistence with recovery, compression and eviction in the live path, storage targets with local fallback, the dashboard and Prometheus metrics, and the lifeboat-cache command.