SGLang HiCache Backend

SGLang’s hierarchical cache (HiCache) keeps KV pages in three levels. LifeboatCache is the third:

Level	Holds	Sized by
GPU	the radix (prefix) cache in GPU memory	SGLang’s `--mem-fraction-static`
Host	a host-RAM page pool	`--hicache-size` / `--hicache-ratio`, or `LIFEBOAT_SGLANG_HOST_GB`
Storage	LifeboatCache — local NVMe, NFS, ONTAP, S3	`lifeboat.yaml`

SGLang copies pages from the GPU to the host pool and backs them up to storage in the background; when a request’s prefix is found in storage but not in memory, SGLang prefetches those pages into the host pool and from there to the GPU.

What does the backend implement?

LifeboatHiCacheStorage (lifeboat_cache.integrations.sglang_backend) implements SGLang’s storage interface:

Zero-copy page I/O (batch_set_v1 / batch_get_v1, on by default): pages are taken as views of the host pool and encoded on LifeboatCache’s own worker threads, instead of SGLang copying each page out on its backup thread. LIFEBOAT_SGLANG_ZERO_COPY=0 falls back to the copying interface; both store the same bytes.
Hybrid-model state (batch_set_v2 / batch_get_v2): for Mamba / linear-attention hybrids, the recurrent-state pools are persisted losslessly next to the attention KV, each in its own key namespace, so a restored prefix comes back with its state.
Parallel I/O. SGLang hands over up to 128 pages per call; pages are compressed or decompressed and written or read in parallel (engine.io_workers, 8 threads).
Per-rank keys. Each tensor-parallel rank stores its own shard under its own key space; MLA models, whose KV is replicated across ranks, share rank 0’s shard.
Recovery. At startup every rank rebuilds its index from storage in the background, so a restarted server serves prefixes stored before the restart.

How big should the host pool be?

The host pool is where prefetched pages land, so it bounds how much KV can be restored for returning users at the same time. A pool smaller than one returning prompt’s KV yields very few prefetches; size it above the KV of the prompts you expect to return together.

Setting	Effect
default	10% of available RAM, clamped to 2–16 GB per rank
`LIFEBOAT_SGLANG_HOST_GB=64`	64 GB per rank
`LIFEBOAT_SGLANG_HOST_GB=auto`	twice the GPU’s memory, within half of the host RAM left after a 10 GB reserve, split across this host’s ranks
`--hicache-size` / `--hicache-ratio` on the launch	always wins

Large host pools are opt-in: raise the pool in steps and confirm GPU memory headroom and throughput under your peak concurrency before production, since a bigger pool also means more host↔GPU transfer work during prefetches.

Which write policy should I use?

SGLang’s HiCache write policy decides which pages reach the host pool (and therefore storage). LifeboatCache keeps the launch’s policy unless LIFEBOAT_SGLANG_WRITE_POLICY is set:

Policy	Persists	Trade-off
`write_through` (SGLang’s default)	every page, on first sight	a prompt seen once can return from storage later; costs storage bandwidth on traffic that never repeats
`write_through_selective`	a page only after it is hit again	less write traffic; a prompt seen once is not persisted
`write_back`	a page only when the GPU evicts it	least write traffic; least persisted

For “returning users” — a document or session sent once and resumed later — keep write_through. For mostly-unique traffic, write_through_selective avoids writing pages that will never be read.

Why does LifeboatCache set a 64-token page?

SGLang’s default page is one token, and HiCache stores one object per page — a 5,000-token prompt would become 5,000 storage objects. LifeboatCache sets --page-size 64 unless the launch gave one (LIFEBOAT_SGLANG_PAGE_SIZE changes the default). Larger pages mean fewer, larger objects; a prefix match is then counted in whole pages. It also bounds SGLang’s GPU write-back staging buffer to 1,024 tokens (LIFEBOAT_SGLANG_STAGING_TOKENS), which would otherwise grow with the page size and come out of activation memory.

What should I know before restarting a server?

Backups to storage run after a request completes. A long session needs a few seconds to be fully written; if the server stops first, the unwritten tail is simply recomputed after the restart — never served corrupted. Flushing SGLang’s cache (/flush_cache) also drops HiCache’s pending backups, so let writes settle first if you flush before a measurement.

Does it run on Apple silicon?

Yes, via SGLang’s MLX backend (SGLANG_USE_MLX=1), where SGLang has no HiCache. There, LIFEBOAT_CACHE=1 registers LifeboatCache as SGLang’s prefix-cache backend (--radix-cache-backend lifeboat): finished requests’ 64-token chunks are copied out of the MLX KV pool and stored uncompressed by a background writer, and a short prefix match is extended with the chunks LifeboatCache holds. Reads for returning users start when their request arrives. Hybrid models are supported one request at a time (concurrent hybrid requests crash inside SGLang’s MLX runtime).