vLLM Native vs LMCache Mode

LifeboatCache has two vLLM integrations, chosen with engine.vllm_mode. Native mode is the default and the recommendation for new deployments; LMCache mode exists for teams already standardised on LMCache.

What is the difference?

	Native mode (`engine.vllm_mode: native`, default)	LMCache mode (`engine.vllm_mode: lmcache`)
vLLM connector	`OffloadingConnector` + `TieringOffloadingSpec`	`LifeboatConnector` (LMCache’s multiprocess connector with LifeboatCache defaults)
Host-RAM tier	vLLM’s CPU tier in `/dev/shm`, `engine.native_cpu_gb` (20 GB)	LMCache’s L1 in its server process, `engine.lmcache_l1_size_gb` (`auto` = 40% of available RAM, 4–128 GB)
Where LifeboatCache runs	inside vLLM’s scheduler process, as the secondary tier	inside the LMCache server, as its L2 adapter (`LifeboatL2Adapter`)
Extra install	none	`pip install "lifeboatcache[vllm]"` (LMCache 0.5.x)
Hybrid Mamba / linear-attention models	supported — attention KV and recurrent state are offloaded together	standard-attention models only: LMCache does not carry the recurrent state of hybrid models at long context
Sliding-window models (e.g. gpt-oss)	supported; vLLM’s async scheduling is turned off for them (0.1.24, `engine.native_async_scheduling`): with it on, vLLM 0.32’s offload copies raced the model under a full CPU tier and the engine crashed. Costs some throughput; `--async-scheduling` overrides it	—
Stored format	raw blocks	the tier’s compression codec
CPU-only vLLM (Apple silicon, x86 / ARM servers)	supported	not available (LMCache’s server cannot register CPU KV)
Returning users whose KV exceeds the host tier	waits up to `engine.native_promotion_wait_s` (5 s) for room instead of recomputing	a single request larger than about 80% of the L1 cannot be staged

How does native mode work?

vLLM  (GPU KV + Mamba state, prefix cache)
  └─ OffloadingConnector / TieringOffloadingSpec
       └─ CPU primary tier         (/dev/shm, engine.native_cpu_gb)
            └─ secondary tier: LifeboatTier
                 └─ storage tiers: NVMe / NFS / ONTAP / S3 / local fallback
                    + index, eviction, dashboard :9400, Prometheus :9402

vLLM moves blocks between the GPU and its CPU tier for every KV-cache group. LifeboatCache persists what vLLM offloads, keyed by vLLM’s own storage namespace (model, parallel layout, dtype, KV-cache groups), and answers lookups asynchronously on vLLM’s lookup thread. When vLLM’s CPU tier is momentarily full, a request whose blocks are on disk waits for room rather than being recomputed from scratch.

On models whose KV-cache groups all share one block size, offload blocks are 256 tokens (engine.native_block_tokens, auto); vLLM’s own default of 16-token blocks makes one small file per block and stalls under many concurrent users.

How does LMCache mode work?

vLLM  (GPU KV, prefix cache)
  └─ LifeboatConnector  = LMCache MP connector + LifeboatCache defaults
       └─ LMCache MP server (auto-started, localhost:6555)   L1 = host RAM
            └─ L2 adapter: LifeboatL2Adapter
                 └─ storage tiers, compression, index, eviction, dashboard, Prometheus

LIFEBOAT_CACHE=1 with engine.vllm_mode: lmcache starts a local LMCache server with these defaults, all settable in lifeboat.yaml:

Key	Default
`engine.lmcache_host` / `lmcache_port`	`localhost` / `6555`
`engine.lmcache_l1_size_gb`	`auto`
`engine.lmcache_chunk_size`	`256`
`engine.lmcache_eviction_policy`	`LRU`
`engine.lmcache_prefetch_max_in_flight`	`1` — LMCache’s own default of 8 lets concurrent prefetches of long prompts crowd each other out of the L1
`engine.l2_inflight_gb`	`4` — writes accepted but not yet on disk; beyond it, chunks stay in L1 instead of growing memory
`engine.autostart_wait_timeout_s`	`180`
`engine.lmcache_extra_args`	extra `lmcache server` arguments

Or explicitly, with no environment variable:

vllm serve <model> --kv-transfer-config '{"kv_connector":"LifeboatConnector","kv_role":"kv_both"}'

How do I switch modes?

engine:
  vllm_mode: lmcache      # or native (default)

Restart vLLM with LIFEBOAT_CACHE=1. The log names the mode in use: LIFEBOAT_CACHE=1: Lifeboat Cache via LMCache or … via vLLM native offloading.

I already run LMCache — what happens?

LifeboatCache never replaces a connector you configured: an explicit --kv-transfer-config always wins, with or without LIFEBOAT_CACHE=1.

Your setup	With `lifeboatcache` installed
no KV connector	`LIFEBOAT_CACHE=1` adds native mode (or LMCache mode if configured); without it, nothing changes
`LMCacheMPConnector` + your own `lmcache server`	untouched. Add LifeboatCache to your server as an extra L2 adapter (below)
`LifeboatConnector` pointed at a server (`lmcache.mp.host` / `port` set)	connects to it; no second server is started
in-process `LMCacheConnectorV1` (legacy, non-MP)	not supported — use the MP connector

Adding LifeboatCache to an existing LMCache server — --l2-adapter is repeatable, so your existing adapters stay:

lmcache server ... --l2-adapter '{"type":"plugin","module_path":"lifeboat_cache.integrations.lmcache_adapter","class_name":"LifeboatL2Adapter","adapter_params":{"lifeboat_config":"/etc/lifeboat.yaml"}}'

To run one LifeboatCache-backed LMCache server for several vLLM instances, start it with lifeboat-cache server (it reads the engine.lmcache_* settings; extra arguments after it are passed to lmcache server), and point each vLLM’s LifeboatConnector at it.