Skip to content
Docs

Tiers & Storage Backends

LifeboatCache sits below the engine’s host-RAM tier. The engine still moves KV between GPU memory and host RAM; LifeboatCache stores what leaves host RAM on one or more storage tiers and loads it back on a prefix hit.

Which storage tiers are supported?

Section titled “Which storage tiers are supported?”
type For Required keys Optional keys
filesystem any local or mounted directory path capacity_gb, min_free_gb (10), direct_io (true)
nvme local NVMe (Linux; elsewhere it behaves as filesystem) path as filesystem
nfs an NFS export path, and mount (must be a mount point) capacity_gb
ontap NetApp ONTAP over NFS / pNFS mount or path protocol (pnfs, pnfs_rdma), strict_protocol, parallelism (8), rest (management API), qos_policy, prefer_nearest_flexcache
s3 S3 or an S3-compatible object store bucket prefix, region, endpoint_url, multipart_parallelism (8), byte_range_reads (true), capacity_gb
memory tests and development — capacity_gb

Every tier also takes name (unique), enabled (default false — a tier must be switched on) and scope (node, rack, pod, cluster, region, fleet; informational for placement). The nixl and doca_memos types exist in the schema but are not validated on hardware; on a host without those transports they fall back to an in-memory tier.

The s3 tier needs boto3 (pip install "lifeboatcache[s3]") and takes credentials from the standard AWS chain (environment, profile, instance role). An ONTAP rest block takes host, svm, verify_tls and credentials_env — the name of an environment variable holding user:password, so no secret is written into the file.

How do I configure a tier?

Section titled “How do I configure a tier?”

In lifeboat.yaml:

storage:
  tiers:
    - {name: nvme,  type: nvme,  enabled: true, path: /mnt/nvme/lifeboat-kv, capacity_gb: 800}
    - {name: ontap, type: ontap, enabled: true, mount: /mnt/kvcache, protocol: pnfs_rdma}
    - {name: s3,    type: s3,    enabled: true, bucket: kv-archive, region: us-east-1, prefix: prod/}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50}

Order matters. New writes go to the first usable tier in the list; lookups check every usable tier, nearest first. Put the fastest tier first.

What happens when a storage target is unreachable?

Section titled “What happens when a storage target is unreachable?”

Each enabled tier is write-probed at startup — a small object is written, read back and deleted. A tier that fails is skipped with the reason, logged and shown on the dashboard:

tier ontap failed its startup probe (OSError: …); skipping
tier s3 unavailable: RuntimeError: s3 tier 's3' requires boto3 (not installed)

If no persistent tier is left (or none is configured), KV goes to storage.fallback_local on local disk instead of the cache silently becoming memory-only:

no persistent tier available (ontap: …); writing KV to local fallback /home/<you>/.lifeboat_cache/kv

At runtime each tier has a circuit breaker: when more than storage.backpressure.circuit_breaker.error_rate (20%) of its operations fail, it is skipped for open_s (30 s) and then tried again. Reads fall through to the next tier; a read that finds nothing is a miss, and the engine recomputes.

An ONTAP tier asked for protocol: pnfs or pnfs_rdma checks that the mount really is NFSv4.1 (with RDMA). If it is not, it logs … requested but mount is not pNFS-capable (…); continuing with plain NFS/file I/O (fail-safe), or refuses the tier when strict_protocol: true.

How is stored KV protected and recovered?

Section titled “How is stored KV protected and recovered?”
  • Integrity. Each stored block carries a content hash that is verified on every read. A block that fails is logged as corrupt chunk … -> treated as miss and recomputed: a damaged file can cost time, never change a response.
  • Self-describing format. Blocks carry their own header, and keys include the model and engine layout (on vLLM: model, parallel layout, dtype and KV-cache groups), so a block is never served to an incompatible configuration.
  • Recovery after restart. Local tiers keep a SQLite manifest (lifeboat-index.sqlite3) next to the data, so the index is rebuilt in the background in well under a second for 100,000 blocks, with no directory scan. NFS tiers have no manifest and are scanned. The log reports recovered N entries (… MiB) from disk in …s.
  • Never fill the disk. Local tiers evict to keep min_free_gb (10 GB) free on the device, whatever capacity_gb says.

How fast are reads from local disk?

Section titled “How fast are reads from local disk?”

On Linux a block’s first read uses O_DIRECT (no page-cache churn for blocks read once); a block read again goes through the page cache, so hot blocks are then served from RAM. Blocks still in the page cache — for example, just written — are always served from it. Turn O_DIRECT off per tier with direct_io: false (or storage.fallback_local.direct_io: false). NFS tiers never use it.

Can several engine processes share one tier directory?

Section titled “Can several engine processes share one tier directory?”

Yes. For one engine per GPU behind a load balancer, give every process the same tier path and set:

storage:
  shared_lookup: true          # on an index miss, look for a block another process wrote
  shared_evict_grace_s: 30     # a block looked up this recently is never evicted

A user who returns on a different GPU is then restored, not recomputed. With shared_lookup, capacity_gb of a local tier covers the whole directory, and one process at a time evicts (a file lock, taken over if the holder dies). NFS tiers enforce capacity per process.

Sources and references

Section titled “Sources and references”