Skip to content
Docs

Benchmarks

LifeboatCache is benchmarked on agent sessions, the workload where KV reuse matters most: every turn re-sends the whole conversation so far plus what the agent added, so a server that keeps earlier turns’ KV skips most of each prefill. The workload replays public agent traces (sammshen/lmcache-agentic-traces), with and without LifeboatCache, across host-DRAM sizes.

How is the agentic benchmark run?

Section titled “How is the agentic benchmark run?”
  • Sessions: real agent traces from public datasets (SWE-bench Verified, GAIA, WildClawBench), replayed turn by turn. Each turn’s prompt is exactly the previous prompt plus what the agent appended, rendered as plain text, so prefixes repeat as they do in production agents.
  • Concurrency: N agents run at once against the server’s OpenAI-compatible API, each replaying its own sessions.
  • Equal work: requests ignore end-of-sequence, so every configuration generates the same number of output tokens.
  • Isolation: a fresh server with empty caches and tiers for every cell of the grid.
  • The grid: engine × configuration × host-DRAM size × number of agents. “DRAM” is the host-RAM offload tier given to the configuration under test.
  • Reported: mean and P99 time to first token (TTFT), sustained output tokens per second, prefix-cache hit rate, wall-clock time.

The configurations compared on vLLM:

Configuration What it is
vLLM alone vLLM’s GPU prefix cache only; no offloading (DRAM size does not apply)
vLLM + LMCache LMCache with a host-RAM tier of the DRAM size
vLLM + LifeboatCache vLLM’s own offloading with a host-RAM tier of the DRAM size, and LifeboatCache below it on local NVMe (native mode)

What was measured on one RTX PRO 6000?

Section titled “What was measured on one RTX PRO 6000?”

Setup (2026-10-30): 1 × RTX PRO 6000 Blackwell Server Edition (96 GB), AMD EPYC 9554, 1.5 TB RAM, local NVMe, in a rented cloud pod whose /dev/shm was capped at 88 GB (so no 128 GB cells). vLLM 0.32.0, LMCache 0.5.6, torch 2.13.0, LifeboatCache 0.1.23. Model Qwen3-30B-A3B. 10 rounds, 256 output tokens per request, 40,960-token context, gpu-memory-utilization 0.80, KV cache dtype auto (bf16).

Mean TTFT / output tokens per second at 64 concurrent agents:

DRAM vLLM alone vLLM + LMCache vLLM + LifeboatCache 0.1.23
16 GB 25.4 s / 180 25.2 s / 185 13.5 s / 310
32 GB 25.4 s / 180 21.8 s / 211 (run 2: 22.2 / 208) 13.0 s / 323 (run 2: 12.7 / 325)
64 GB 25.4 s / 180 16.6 s / 268 12.8 s / 323

At every DRAM size measured, vLLM + LifeboatCache had the lowest mean time to first token and the highest output throughput of the three, and more DRAM improved both offload configurations. The 32 GB cells were run twice; the second run’s figures are in brackets.

In every configuration, 2, 3 and 6 requests failed at 16, 32 and 64 agents respectively: those sessions grow longer than the 40,960-token context.

What changed in 0.1.23 for this workload?

Section titled “What changed in 0.1.23 for this workload?”

This grid is where the 0.1.23 default was found: vLLM’s native offloading moves 16-token blocks by default, which made one small file per block. LifeboatCache 0.1.23 offloads 256-token blocks on models whose KV-cache groups share one block size (engine.native_block_tokens, auto). The table above is 0.1.23.

What was measured on four RTX PRO 6000?

Section titled “What was measured on four RTX PRO 6000?”

Setup (2026-10-30): 4 × RTX PRO 6000 Blackwell Server Edition (96 GB each), Intel Xeon Platinum 8559C, 1 TB RAM, 2 × local NVMe in RAID 0. vLLM 0.32.0, LMCache 0.5.6, LifeboatCache 0.1.23. Model Qwen3-235B-A22B-FP8 with tensor parallelism 4 (the GPUs hold 387,536 tokens of KV, about ten full-length sessions). Same workload settings as above: 10 rounds per agent and a 256-token output cap. DRAM is the whole server’s host-RAM tier, equal for both caches; 128 GB is 32 GB per GPU.

Mean TTFT / output tokens per second, 128 GB of DRAM:

Agents vLLM alone vLLM + LMCache vLLM + LifeboatCache 0.1.23
32 0.90 s / 204 1.10 s / 206 0.94 s / 204
64 26.7 s / 94 17.1 s / 148 (run 2: 17.0 / 146) 9.1 s / 219 (run 2: 9.2 / 219)
128 137.5 s / 70 113.5 s / 87 42.7 s / 208

At 32 agents the GPUs hold most sessions and all three are equal: the cache adds no overhead. At 64 agents LifeboatCache served 75% of prompt tokens from cache (LMCache 53%, vLLM alone 12%); at 128 agents it delivered 3.0x the output throughput of vLLM alone.

Energy, latency over the session, and capacity within a latency target (128 GB of DRAM):

vLLM alone vLLM + LMCache vLLM + LifeboatCache 0.1.23
Output tokens per joule (GPU), 64 / 128 agents 0.071 / 0.053 0.113 / 0.065 0.171 / 0.159
Mean TTFT, round 1 → round 10, 64 agents 2.1 → 29.5 s 2.0 → 20.2 s 2.1 → 8.9 s
Mean TTFT, round 1 → round 10, 128 agents 10.7 → 182.9 s 10.8 → 154.2 s 10.8 → 45.4 s
Most agents with p99 TTFT within 30 s (of 32 / 64 / 128 tested) 32 32 64
Wall-clock for the same 640 / 1,280 requests 12.9 / 34.8 min 8.1 / 28.3 min 5.5 / 11.8 min

GPU energy is from nvidia-smi over each run; CPU package energy was not readable on this cloud server. LifeboatCache’s 128-agent figures use the 2 TB NVMe tier described below.

64 agents across DRAM sizes:

DRAM (per server) vLLM + LMCache vLLM + LifeboatCache 0.1.23
64 GB 24.8 s / 104 9.2 s / 219
128 GB 17.1 s / 148 9.1 s / 219
256 GB 10.0 s / 217 9.1 s / 220

At 64 agents the workload’s reuse tops out at about 75% of prompt tokens; LifeboatCache keeps on local NVMe what does not fit in DRAM.

Two things to size when you reproduce this:

  • The NVMe tier. 128 agents’ KV on a 235B model is far more than the 250 GB the benchmark launcher gives LifeboatCache by default: with 250 GB, LifeboatCache at 128 agents gave 80.3 s / 119 tok/s (50% cached); with 2 TB, the result in the table (79% cached). Set fallback_local.capacity_gb (or the tier’s capacity_gb) to the space you have — see Eviction & capacity.
  • vLLM’s CPU tier above half of RAM. At 512 GB on this 1 TB server, vLLM 0.32 could not pin its CPU offload tier on three of four workers during startup and copied through pageable memory, so native mode ran slower (16.1 s / 123 tok/s). 256 GB pinned and ran at full speed. Keep engine.native_cpu_gb at or below about a quarter of host RAM with four GPUs until this is resolved.

DeepGEMM was disabled (VLLM_USE_DEEP_GEMM=0) in every configuration of this run. Per-request data, per-cell summaries and the tools are in IterateAI/LifeboatCacheTest.

What about long agent sessions at 256 and 512 agents?

Section titled “What about long agent sessions at 256 and 512 agents?”

Setup (2026-10-31): the same 4 × RTX PRO 6000 server and model, with 25 rounds per agent, up to 512 output tokens per round and a 40,192-token context: sessions grow to tens of thousands of tokens, far beyond what the GPUs hold. LifeboatCache 0.1.23 with 256 GB of host DRAM and a 3 TB NVMe tier.

Agents Mean TTFT P99 TTFT Output tok/s Prompt tokens from cache Requests completed
256 158 s 242 s 222.6 89% 6,391 of 6,400
512 598 s 1,365 s 122.4 90% 12,789 of 12,800

The few requests that did not complete were sessions longer than the context. Time to first token includes queueing: 256 to 512 agents share four GPUs.

DRAM at 256 agents: with 64 GB of host DRAM instead of 256 GB (same NVMe tier), LifeboatCache gave 199.0 tok/s and 178 s mean TTFT (85% from cache); 256 GB gave 222.6 tok/s and 158 s (89%) — 12% more throughput from the extra DRAM.

Disk: the two local NVMe drives (RAID 0) read at 12–15 GB/s and write at 5.4 GB/s (fio); at 512 agents they ran at about 25% utilisation, so the NVMe tier was not the limit. Size capacity_gb for the working set: at 512 agents on this model LifeboatCache’s tier reached about 3 TB.

What about gpt-oss-20b on one GPU?

Section titled “What about gpt-oss-20b on one GPU?”

Setup (2026-10-31): one RTX PRO 6000 Blackwell (Workstation Edition), gpt-oss-20b (MXFP4), 128,000-token context, 25 rounds per agent, up to 512 output tokens per round, 24 GB of host DRAM for both caches. LifeboatCache 0.1.23.

Agents Configuration Mean TTFT Output tok/s Prompt tokens from cache Output tokens per joule (GPU)
128 vLLM alone 0.49 s 1,117 80% 1.90
128 vLLM + LMCache 0.64 s 1,144 82% 1.95
128 vLLM + LifeboatCache 0.1.23 0.44 s 1,425 89% 2.55
256 vLLM alone 70.7 s 342 7% 0.57
256 vLLM + LMCache 68.7 s 349 9% 0.58
256 vLLM + LifeboatCache 0.1.24 54.4 s 349 40% 0.74

At 128 agents LifeboatCache gave 28% more throughput than vLLM alone and 34% more output tokens per joule. At 256 agents it gave 23% lower mean time to first token and 30% more tokens per joule at equal throughput. On these sliding-window models LifeboatCache 0.1.24 turns vLLM’s async scheduling off: with it on, vLLM 0.32’s offload copies raced the model under a full CPU tier and the engine crashed (engine.native_async_scheduling). gpt-oss mixes sliding-window and full attention; vLLM keeps only the window for the sliding layers, so one GPU holds about 2.3 million tokens of its KV.

What was measured on AMD Instinct MI210?

Section titled “What was measured on AMD Instinct MI210?”

Setup (2026-11-01 to 11-03): one AMD Instinct MI210 (64 GB, ROCm 7.2), AMD EPYC 9555P, 250 GB RAM, local NVMe. Model Qwen2.5-7B-Instruct (bf16), 32,768-token context, 10 rounds per agent, up to 512 output tokens per round. Every offload configuration gets the same 64 GB of host DRAM. vLLM 0.32.0 (ROCm image), SGLang 0.5.19 and Lifeboat 2.2.62. LifeboatCache 0.1.24 / 0.1.25, and 0.1.28 for LMCache + LifeboatCache.

vLLM, every KV offload option — mean TTFT / output tokens per second:

vLLM with 128 agents 256 agents
nothing (GPU prefix cache only) 260.0 s / 45 698.1 s / 41
vLLM’s own offloading (CPU + filesystem tier) 55.3 s / 148 172.7 s / 152
LMCache, DRAM 158.7 s / 66 636.9 s / 45
LMCache, DRAM + NVMe 65.3 s / 128 350.1 s / 78
LMCache + LifeboatCache as its storage tier 68.4 s / 123 325.7 s / 85
LifeboatCache (native mode, DRAM + NVMe) 55.0 s / 147 179.9 s / 145
LifeboatCache with OSCAR + EpiCache 55.6 s / 147 182.2 s / 143

vLLM + LifeboatCache gave 3.2x the output throughput of vLLM alone at 128 agents and 3.6x at 256, with 79% and 83% of prompt tokens served from cache. vLLM’s own CPU + filesystem offloading was level with it on this model (within 1% at 128 agents, 5% ahead at 256). As LMCache’s storage tier, LifeboatCache was 4% behind LMCache’s own NVMe tier at 128 agents and 9% ahead at 256. LMCache with DRAM alone could not hold 256 agents’ sessions (almost nothing served from cache).

SGLang and Lifeboat:

Engine Agents No offload + LifeboatCache + LifeboatCache, OSCAR + EpiCache
SGLang 0.5.19 128 93.3 s / 71 84.9 s / 84 84.9 s / 84
Lifeboat 2.2.62 128 93.2 s / 72 84.2 s / 86 84.5 s / 86
SGLang 0.5.19 256 410.5 s / 58 402.6 s / 60 412.5 s / 58
Lifeboat 2.2.62 256 410.6 s / 58 404.3 s / 59 410.5 s / 59

At 128 agents LifeboatCache gave SGLang and Lifeboat 19–20% more throughput. That gain comes from SGLang’s host-DRAM tier: SGLang 0.5.19 writes pages to a storage tier but reads almost none of them back, with LifeboatCache or with SGLang’s own file backend, so at 256 agents, once the sessions outgrow DRAM, there is no gain. The fix belongs in the engine, not in LifeboatCache.

OSCAR and EpiCache: on SGLang and Lifeboat, OSCAR halved the bytes LifeboatCache stored at the same speed; vLLM’s native mode stores raw bytes, so it does not apply there. With a 40 GB NVMe tier at 128 agents, so that eviction runs, EpiCache was level with LRU (109 vs 110 tok/s).

Running LMCache on an MI210: the LMCache 0.5.3 in vLLM’s ROCm image has GPU kernels for gfx942 and gfx950 (MI300 series) only. On an MI210 (gfx90a) its KV transfer crashes and vLLM keeps serving with no cache. The LMCache numbers above use the same LMCache rebuilt for the MI210 (PYTORCH_ROCM_ARCH=gfx90a pip install --no-build-isolation --no-binary lmcache lmcache==0.5.3). LifeboatCache 0.1.27 and later refuse to start in LMCache mode on a build without kernels for the GPU, rather than run uncached.

How does this compare with the published industry study?

Section titled “How does this compare with the published industry study?”

Metrum AI’s public study Streamlined Agentic Inference with KV Cache Offload (September 2026, HPE ProLiant DL380a Gen12 with Intel Xeon 6) ran the same model, Qwen3-235B-A22B-FP8 on 4 × RTX PRO 6000, with agent sessions of 25 rounds and 512 output tokens, using vLLM 0.22 with LMCache 0.4.6 for the offload tiers. Its own figures, with LifeboatCache’s from the long-session run above:

256 agents: mean TTFT / output tok/s 512 agents: mean TTFT / output tok/s Output tokens per joule (GPU), 256 agents
Study: GPU only 839.5 s / 47.0 1,725.8 s / 44.9 0.033
Study: best offload (LMCache, DRAM + NVMe) 309.4 s / 124.0 688.9 s / 111.2 0.089
vLLM + LifeboatCache 0.1.23 158.0 s / 222.6 598.2 s / 122.4 0.173

This is not a head-to-head run. The study used vLLM 0.22, LMCache 0.4.6 and a 2 TB host (1 TB given to LMCache); LifeboatCache ran on vLLM 0.32 with a 1 TB host, 256 GB of DRAM and a 3 TB NVMe tier, so part of the difference can come from the newer vLLM. The study’s figures are its own and were not reproduced here. Its main finding, about 2.7x throughput from offloading KV to DRAM and NVMe instead of serving from the GPU alone, matches what is measured above.

What about Apple silicon?

Section titled “What about Apple silicon?”

Setup: one Apple M4 Max (64 GB unified memory), Qwen3-0.6B, LifeboatCache 0.1.14 / 0.1.15, measured with the LifeboatCacheTest verifier: users whose context no longer fits in memory return with a new question, then the server restarts. A functional check on a small model, not a capacity benchmark.

Engine Returning user TTFT, no offload With LifeboatCache After a server restart Peak output tok/s, off → on
SGLang (MLX) 7.97 s 1.48 s (99.7% from cache) 0.88 s 151 → 123
vLLM 0.32 (CPU backend) 49.8 s 1.11 s (98.6% from cache) 0.55 s 26.6 → 26.6

A returning session was 5.4x faster on SGLang and 45x faster on vLLM’s CPU backend. Peak throughput was unchanged on vLLM and 19% lower on SGLang’s MLX backend with the cache on.

How do I run a comparable test on my own hardware?

Section titled “How do I run a comparable test on my own hardware?”

Replay real multi-turn sessions rather than independent prompts, and keep three things fixed across the configurations you compare: the same sessions, the same output length with end-of-sequence ignored, and empty caches and tiers at the start of every cell. Size the DRAM tier as you would in production — it is the variable that decides how many returning sessions can be staged at once — and report mean and P99 time to first token alongside output tokens per second. The tools used here — the session replay, the grid driver and the server launcher — are public in IterateAI/LifeboatCacheTest under agentic/, with the raw results under reports/. For help setting up a comparison, contact Iterate.ai through iterate.ai/contact.

Sources and references

Section titled “Sources and references”