Quickstart on SGLang

On SGLang, LifeboatCache is a HiCache storage backend. SGLang’s hierarchical cache keeps KV pages in GPU memory and in a host-RAM pool; LifeboatCache is the storage layer below that pool, so pages survive eviction and restarts.

How do I start SGLang with LifeboatCache?

pip install lifeboatcache          # in SGLang's environment
LIFEBOAT_CACHE=1 python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B --port 30000

sglang serve works the same way. The plugin runs before SGLang parses its arguments and fills in only what the launch left unset.

What does LIFEBOAT_CACHE=1 change in the launch?

Server argument	Set to	Unless
`--enable-hierarchical-cache`	on	—
`--hicache-storage-backend`	`dynamic`, loading `LifeboatHiCacheStorage`	the launch already chose a storage backend — then LifeboatCache steps aside
`--page-size`	64 tokens (`LIFEBOAT_SGLANG_PAGE_SIZE`)	`--page-size` was given
`--hicache-size` (host pool, GB per rank)	10% of available RAM, clamped to 2–16 GB (`LIFEBOAT_SGLANG_HOST_GB`)	`--hicache-size` or `--hicache-ratio` was given
`--hicache-write-policy`	unchanged	`LIFEBOAT_SGLANG_WRITE_POLICY` is set
zero-copy page interface	on (`LIFEBOAT_SGLANG_ZERO_COPY=0` turns it off)	—
`--mamba-radix-cache-strategy` (hybrid models)	`extra_buffer_lazy`	the launch set a strategy
radix cache, if the launch disabled it	re-enabled (HiCache needs it)	`LIFEBOAT_CACHE_FORCE_RADIX=0` — then LifeboatCache stays off

Why 64-token pages: SGLang’s default page is one token, which would make every token its own storage object. Why a small host pool: LifeboatCache persists to storage, so the host pool only has to stage pages; HiCache’s own default (twice the GPU KV pool) can exceed host RAM on a large GPU.

The same launch with explicit flags, no plugin involved:

python -m sglang.launch_server --model-path Qwen/Qwen3-30B-A3B --enable-hierarchical-cache \
  --page-size 64 \
  --hicache-storage-backend dynamic \
  --hicache-storage-backend-extra-config '{"backend_name":"lifeboat","module_path":"lifeboat_cache.integrations.sglang_backend","class_name":"LifeboatHiCacheStorage","interface_v1":1}'

Add "lifeboat_config": "/etc/lifeboat.yaml" to the extra config to use a specific configuration file.

How do I know it attached?

LIFEBOAT_CACHE=1: HiCache enabled with the Lifeboat storage backend (page_size=64)
Lifeboat Cache dashboard: http://127.0.0.1:9400/

The dashboard is served by the tensor-parallel rank 0 process; other ranks log dashboard port 9400 unavailable (…); another process (e.g. a different TP rank) is probably serving it, which is expected.

If you see SGLang plugin hooks unavailable; pass the --hicache-* flags explicitly, your SGLang build has no plugin hook for Python-API launches — use the explicit flags above.

How do I verify a restore after a restart?

Save this as probe.py; it sends one long prompt to SGLang’s native /generate endpoint and prints the time to the first (and only) token:

import json, time, urllib.request
doc = " ".join(f"Clause {i}: the supplier delivers parts within ten days." for i in range(900))
body = {"text": doc + "\nWhich clause covers delivery?", "sampling_params": {"max_new_tokens": 1}}
req = urllib.request.Request("http://localhost:30000/generate", data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t = time.time(); urllib.request.urlopen(req).read(); print(f"{time.time() - t:.2f} s")

python3 probe.py          # 1. first visit: full prefill
# 2. wait a few seconds for the background write, then restart SGLang with the same command
python3 probe.py          # 3. returning visit: pages restored from LifeboatCache
lifeboat-cache status     # hit rate, loads served, bytes on each tier

SGLang writes pages to storage in the background after a request: allow it a few seconds for long prompts before a planned restart. A tail that was not yet written is recomputed after the restart, never corrupted.

Which knobs matter on SGLang?

Variable	Default	Effect
`LIFEBOAT_SGLANG_HOST_GB`	10% of available RAM, 2–16 GB per rank	HiCache host pool size per rank. A number sets it; `auto` sizes it from the GPU and host RAM (opt-in — validate under your peak load first)
`LIFEBOAT_SGLANG_PAGE_SIZE`	`64`	Tokens per page, i.e. per storage object
`LIFEBOAT_SGLANG_WRITE_POLICY`	launch’s own	`write_through`, `write_through_selective` or `write_back`. `write_through` persists every page on first sight, which is what lets a prompt seen once come back later
`LIFEBOAT_SGLANG_ZERO_COPY`	`1`	`0` uses SGLang’s copying page interface
`LIFEBOAT_SGLANG_STAGING_TOKENS`	`1024`	Bound on SGLang’s GPU write-back staging buffer
`LIFEBOAT_CACHE_FORCE_RADIX`	`1`	`0` keeps a launch’s disabled radix cache disabled (LifeboatCache then stays off)

The size of the host pool bounds how much KV can be staged for returning users at once: a pool smaller than one returning prompt’s KV gives few prefetches. See SGLang HiCache backend for sizing and the storage-side settings in lifeboat.yaml.

Does it work on Apple silicon?

Yes, through SGLang’s MLX backend: with SGLANG_USE_MLX=1 and LIFEBOAT_CACHE=1, the plugin selects LifeboatCache as SGLang’s prefix-cache backend (--radix-cache-backend lifeboat) instead of HiCache. Full-attention and hybrid Mamba / linear-attention models are supported; concurrent requests on hybrid models crash inside SGLang’s MLX runtime, so run hybrid models one request at a time there. LIFEBOAT_MLX_PREFETCH_GB (default 8, 0 = off) caps the host memory used to prefetch returning users’ chunks.