Skip to content
Docs

Quickstart on Lifeboat

Lifeboat is Iterate.ai’s inference platform; LifeboatCache is a separate package that adds a persistent storage tier below Lifeboat’s hierarchical KV cache (its GPU and host-RAM tiers, described in KV Cache Tiering). With it, a server’s evicted prompts are kept on local NVMe or shared storage and survive a server restart.

How is it different from Lifeboat’s own storage backends?

Section titled “How is it different from Lifeboat’s own storage backends?”

Lifeboat’s KV Cache Tiering already offers File and Arena storage backends, configured per server in the console. LifeboatCache replaces that storage layer with its own: storage targets with automatic fallback (local disk, NFS, NetApp ONTAP, S3), per-tier compression, a whole-server capacity budget, a dashboard and Prometheus metrics.

The two do not stack. If a server’s Storage backend is set in the console, that choice wins and LifeboatCache stays off for that server; leave it unset to use LifeboatCache.

How do I install it into the Lifeboat container?

Section titled “How do I install it into the Lifeboat container?”

The Lifeboat image’s root filesystem is read-only, so the package is installed into a host directory and mounted.

1. Install the package into a host directory (it is a pure-Python wheel; the host needs Python 3.10 or newer only to run pip):

Terminal window
sudo mkdir -p /opt/lifeboatcache /mnt/nvme/lifeboatcache
sudo pip install --target /opt/lifeboatcache --no-deps "lifeboatcache==0.1.23"

--no-deps is deliberate: the image already provides the libraries LifeboatCache needs.

2. Mount it, plus a persistent directory for the KV store, and set three variables. As a Compose override next to your Lifeboat docker-compose.yml:

docker-compose.override.yml
services:
  lifeboat:                       # your Lifeboat service name
    environment:
      LIFEBOAT_CACHE: "1"
      PYTHONPATH: /opt/lifeboatcache
      LIFEBOAT_CACHE_HOME: /lifeboatcache
    volumes:
      - /opt/lifeboatcache:/opt/lifeboatcache:ro
      - /mnt/nvme/lifeboatcache:/lifeboatcache

LIFEBOAT_CACHE_HOME is where LifeboatCache keeps its configuration file (lifeboat.yaml) and, by default, the KV store (kv/). Put it on fast local storage and keep it across container recreates — that is what makes restores after a restart possible.

3. Recreate the container:

Terminal window
docker compose up -d

How do I configure it?

Section titled “How do I configure it?”

Optionally write /mnt/nvme/lifeboatcache/lifeboat.yaml on the host. A minimal one with a 500 GB local budget:

storage:
  fallback_local: {enabled: true, path: /lifeboatcache/kv, capacity_gb: 500}

Without a file, the defaults apply: /lifeboatcache/kv, 50 GB, fp8 compression. Every key is in the Configuration reference; shared storage is in Tiers & storage backends.

In the console, a server’s Hierarchical KV Cache settings are still honoured: Ratio or Size (GB) set the host-RAM tier and Write policy is kept as chosen. Leave Storage backend unset. Note that write_through_selective copies a block only after it has been read again, so a prompt seen once is not persisted; write_through persists everything.

How do I verify it attached?

Section titled “How do I verify it attached?”

Start (or restart) a server from the console, then check its log for:

LIFEBOAT_CACHE=1: HiCache enabled with the Lifeboat storage backend (page_size=64)
Lifeboat Cache dashboard: http://127.0.0.1:9400/

and ask LifeboatCache directly, inside the container:

Terminal window
docker exec <lifeboat-container> python -m lifeboat_cache.config.cli status

To reach the dashboard from a browser, either tunnel to the host and run the command above, or set dashboard.host: 0.0.0.0 and dashboard.auth_token in lifeboat.yaml and publish port 9400. See Dashboard & telemetry.

What about hybrid models whose prefix cache Lifeboat turns off?

Section titled “What about hybrid models whose prefix cache Lifeboat turns off?”

For some hybrid (Mamba / linear-attention) model families Lifeboat starts the server with the prefix cache disabled, and a storage tier needs the prefix cache. With LIFEBOAT_CACHE=1, LifeboatCache turns it back on for that server and selects the memory-saving state-buffer strategy, logging:

LIFEBOAT_CACHE=1: re-enabling the radix cache that the launch disabled, for HiCache (set LIFEBOAT_CACHE_FORCE_RADIX=0 to keep it off)

This combination was validated with Lifeboat 2.2.61 and a hybrid model under concurrent multi-round sessions, a stop with writes in flight, and restores after a restart. To keep Lifeboat’s choice instead, set LIFEBOAT_CACHE_FORCE_RADIX: "0"; LifeboatCache then stays off for those servers.

How do I remove it?

Section titled “How do I remove it?”

Delete the three environment variables and the two volumes from the override and run docker compose up -d. Lifeboat starts exactly as before; the KV directory on the host can be deleted.

Sources and references

Section titled “Sources and references”