Skip to content
Docs

Eviction & Capacity

A storage tier holds what you give it and no more. LifeboatCache enforces a capacity budget per tier and evicts the blocks least likely to be reused when the budget is reached; an evicted block is simply a miss next time, and the engine recomputes it.

What does capacity_gb count?

Section titled “What does capacity_gb count?”

capacity_gb is the most stored KV (after compression) a tier may hold, in GB:

storage:
  tiers:
    - {name: nvme, type: nvme, enabled: true, path: /mnt/nvme/lifeboat-kv, capacity_gb: 800, min_free_gb: 20}
  fallback_local: {enabled: true, path: ~/.lifeboat_cache/kv, capacity_gb: 50, min_free_gb: 10}
Setting Default Meaning
capacity_gb on a configured tier 0 0 = no fixed budget: the tier is bounded only by the free-disk reserve. Set it.
storage.fallback_local.capacity_gb 50 budget of the automatic local tier
min_free_gb 10 local tiers evict to keep this much free on the device, whatever capacity_gb says

Eviction starts when a tier passes 95% of its effective budget and brings it back to 85%, so it runs in batches rather than on every write.

Is the budget per server or per GPU?

Section titled “Is the budget per server or per GPU?”

Per server. One capacity_gb is the budget for the whole inference server, however many GPUs it uses:

Deployment How the budget is applied
vLLM (native or LMCache mode) one LifeboatCache store per server — in vLLM’s scheduler process (native) or the LMCache server (LMCache mode) — so the budget applies once
SGLang / Lifeboat with tensor or pipeline parallelism each rank runs its own store over the same directory and gets 1/(TP × PP) of the budget; ranks write symmetric shards, so the server as a whole stays within capacity_gb
SGLang with an MLA model (KV replicated across ranks) all ranks share one shard and the budget is not divided
several engine processes sharing one directory (storage.shared_lookup: true) capacity_gb covers the whole directory, enforced by one process at a time under a file lock

Releases before 0.1.21 enforced the full budget on every SGLang rank, which could store up to TP × the configured amount — upgrade if you run SGLang or Lifeboat with tensor parallelism.

Which eviction policy should I use?

Section titled “Which eviction policy should I use?”

Set eviction.l1_policy:

Policy Evicts Good for
lru least recently used block simple and predictable; the default when no configuration file exists
arc adaptive mix of recency and frequency scan-heavy traffic (many one-off long prompts mixed with reused ones)
isolated_lru LRU per tenant multi-tenant isolation
episodic lowest score: last access + a decayed bonus for each reuse, trimming each prompt chain tail-first agent and multi-turn traffic; the schema default when a configuration file omits eviction
predictive by predicted next access experimental

Tail-first matters for prefix caches: the head of a prompt chain serves every later block, so evicting from the tail keeps the reusable part longest. In trace simulation through the real store, the score-based policy beat LRU on coding-agent and scan-mix traces at 10–50% capacity and was level on topic-switching traces.

Tuning keys for episodic, both in seconds and tuned at roughly one request per second — scale them with your request rate:

eviction:
  l1_policy: episodic
  episodic:
    freq_weight_s: 10.0       # how much one reuse is worth, in seconds of recency
    freq_half_life_s: 50.0    # how fast that bonus decays
    topic_ordering: false     # evict the coldest topic first (opt-in; lost hit rate to LRU in simulation)

How do I size the tiers?

Section titled “How do I size the tiers?”
  • Host RAM tier (the engine’s): above the KV of the prompts that return at the same time. Everything restored from disk passes through it. On vLLM this is engine.native_cpu_gb (20 GB default); on SGLang the HiCache host pool (LIFEBOAT_SGLANG_HOST_GB).
  • Storage tier: above your working set — (number of distinct long prompts you expect to come back) × (their KV size). A cache smaller than its working set thrashes and returns little.

KV size per token depends on the model, KV dtype and parallel layout; the dashboard’s bytes and chunk counts after a known prompt give the real figure for your deployment.

How do I watch eviction?

Section titled “How do I watch eviction?”

The dashboard shows evictions, bytes per tier and the capacity line; Prometheus exposes lifeboat_evictions_total{tenant,policy}, lifeboat_tier_bytes{tier} and lifeboat_capacity_bytes. A steadily climbing eviction count with a falling hit rate means the tier is smaller than the working set. See Dashboard & telemetry.

Sources and references

Section titled “Sources and references”