Multi-GPU & Tensor Parallelism
LifeboatCache follows the engine’s own process layout, so its behaviour under tensor (TP) and
pipeline (PP) parallelism differs between vLLM and SGLang. In both, capacity_gb is a budget
for the whole server.
How does it work with tensor parallelism on vLLM?
Section titled “How does it work with tensor parallelism on vLLM?”vLLM’s offloading runs its tiers from the scheduler side: the CPU tier is one shared memory
region that holds every worker’s share of each block, and LifeboatCache is created once, as its
secondary tier, in vLLM’s scheduler process. So on a TP=2, 4 or 8 vLLM server there is one
LifeboatCache store, one index and one dashboard, and capacity_gb applies once.
Keys carry vLLM’s storage namespace — model, parallel layout, dtype and KV-cache groups — so blocks written by a TP=2 server are never served to a TP=4 server of the same model; that server simply misses and recomputes. In LMCache mode the store lives in the LMCache server process, also once per server.
How does it work with tensor parallelism on SGLang and Lifeboat?
Section titled “How does it work with tensor parallelism on SGLang and Lifeboat?”Each SGLang rank is its own process with its own LifeboatCache store, all over the same directory:
| Aspect | Behaviour |
|---|---|
| Keys | per rank (tp_rank + tp_size × pp_rank), so each rank stores and restores its own shard |
| MLA models (KV replicated across ranks) | every rank uses rank 0’s shard; written once |
capacity_gb |
each rank enforces 1/(TP × PP) of it (MLA: not divided), so the server stays within the budget |
Host pool (LIFEBOAT_SGLANG_HOST_GB, --hicache-size) |
per rank; auto splits its budget across the ranks on the host |
Dashboard and /metrics |
served by rank 0; other ranks log dashboard port 9400 unavailable (…); another process (e.g. a different TP rank) is probably serving it |
| Recovery at startup | every rank rebuilds its own shard’s index in the background |
Because the dashboard is on rank 0, its numbers describe rank 0’s shard. Ranks write symmetric
shards, so rank 0’s hit rate is representative; for bytes on disk, look at the tier directory
or lifeboat_tier_bytes × ranks.
What about several single-GPU servers behind a load balancer?
Section titled “What about several single-GPU servers behind a load balancer?”A common layout is one engine per GPU with a router in front. A user who returns may land on a different GPU. Give every engine on the node the same tier path and turn on shared lookups:
storage: shared_lookup: true shared_evict_grace_s: 30 fallback_local: {enabled: true, path: /mnt/nvme/lifeboat-kv, capacity_gb: 2000}On an index miss, each process looks for a block another process wrote after it started (one
file open per missed block), so the returning user is restored rather than recomputed.
capacity_gb then covers the whole directory; one process at a time evicts, under a file
lock that another process takes over if the holder dies, and a block looked up in the last
shared_evict_grace_s seconds is never evicted under the load that follows. Each process still
serves its own dashboard, so give each a distinct dashboard.port (for example with
LIFEBOAT__dashboard__port=9401) and telemetry.prometheus.port.
Prefix-aware routing — sending a returning user back to the GPU that already holds the prefix in memory — is complementary: Lifeboat’s load balancer offers a sticky-session algorithm for exactly that (Load-Balancing Algorithms).
Can several nodes share one cache?
Section titled “Can several nodes share one cache?”The shared-directory mode above is built and validated for processes on one node. Pointing
several nodes at the same NFS or ONTAP mount with shared_lookup: true lets a node find blocks
another node wrote, but NFS tiers keep no manifest and enforce capacity per process, so give
each node its own budget share. The fleet controller (lifeboat-controld, the control_plane
settings) adds membership, placement and configuration push across nodes; it is off by default,
and single-node deployments do not need it.