KV Cache Tiering

Reusing a prompt’s key-value cache is the single largest saving available to a serving deployment, and the limit on it is GPU memory. Tiering moves evicted blocks to host RAM — and below that to storage — so a returning prefix is re-loaded rather than re-computed.

This matters for exactly the workloads people actually run: a long system prompt, a tool manifest, a retrieved document, a multi-turn conversation. All of them send the same leading tokens again and again.

What it is worth

Measured on an RTX PRO 6000 Blackwell. The GPU cache is pinned to the same size in both rows, so the only difference is whether a host tier exists behind it. Twelve distinct 3,000-token prompts are cycled — more than the GPU cache can hold, which is the ordinary condition on a busy server.

Tensor engine	cache reuse	response time
GPU cache only	0.14%	82.1 ms
+ host-RAM tier	99.73%	32.9 ms

The GGUF engine has the same capability under a different name, and it is on by default at 8 GB:

GGUF engine	prefill	cache reuse
host cache disabled	250.1 ms	0.8%
host cache enabled	6.1 ms	99.9%

That is 40x less prefill work for a repeated 8,758-token prompt.

Turning it on

Tensor engine. On the server’s Advanced settings, tick Hierarchical KV Cache. The knobs that appear are optional; the defaults are reasonable.

• Ratio — the host tier as a multiple of the GPU cache. 2.0 triples total capacity.
• Size (GB) — an absolute host budget instead of a ratio.
• Write policy — write_through_selective (the default) copies only blocks that have been read again; write_through copies everything for the highest hit rate; write_back copies only on eviction for the least traffic.
• Storage backend — adds a third tier below host RAM. File writes to local disk or any mounted filesystem.

GGUF engine. Set GGUF Host Cache (MB). The engine’s own default is 8192 MB, which is sized for a laptop; on a server this is the highest-value setting on the page, because it decides how many distinct prompts stay resident. -1 means no limit, 0 disables it.

Size the tier above your working set, or it does nothing

This is the one way to get no benefit from a correctly enabled feature. Measured on the same rig, with the host tier deliberately set below the set of prompts in use:

host tier	cache reuse
sized below the working set	0.15%
sized above it	99.73%

The tier is a cache, so a cache smaller than the thing being cached thrashes and returns nothing. Estimate your working set as (number of distinct long prompts) x (their length) and give the tier more than that. Host RAM is usually plentiful — a server with 500 GB of RAM against 96 GB of VRAM has a great deal of room — so err high.

A third tier on storage

Setting a Storage backend puts a third tier below host RAM — local disk, or any mounted filesystem. Measured on the same rig, with the host tier deliberately too small so blocks reach storage:

	cache reuse	response time
GPU + host RAM	0.15%	37.9 ms
+ file storage	49.4%	29.5 ms

Choose the backend by where the files are going

The File backend writes one file per cache key. That run produced 36,732 files for 451 MB — about 12 KB each, so a terabyte of cached prompts is on the order of 80 million files. On local NVMe that is fine. On a shared filesystem it is the binding cost: every write is a create plus a rename, every lookup is a round trip, and the directory itself becomes the problem long before the capacity does.

The Arena backend stores the same data in a small number of large files and answers lookups from memory. Same 36,732 pages, same sizes, both layouts measured in one run on a local SSD:

	files written	write	read	lookups
File	36,732	5.84 s	0.33 s	0.089 s
Arena	3	0.64 s	0.10 s	0.005 s

And end to end, with a server actually handling requests through the cache on an AMD Instinct MI210 — same model, same prompts, cold store on both arms, two runs each:

	cache reuse	files on disk	size
File	52.8% / 51.4%	78,438	975.7 MB
Arena	41.2% / 58.9%	16	973.6 MB

Same bytes to within 0.2%, and 4,902 times fewer objects — identical in both runs.

Cache effectiveness is unchanged, which is what should happen: this changes how the data is laid out on disk, not what is kept or for how long. The two layouts’ results overlap — Arena’s own two runs differ by more than the two layouts differ from each other — so on this evidence neither is better at hitting the cache.

Both sets of figures are from local storage, where a file operation is nearly free. On a network filesystem each one is a round trip, so the gap there is wider than these tables show, not narrower.

Point it at your storage with Extra Config:

{"arena_dir": "/mnt/kv", "capacity_bytes": 549755813888}

capacity_bytes is the total on-disk budget. When it is reached, the least-recently-used extent is removed whole — so set it to what you are willing to give the cache, not to the size of the filesystem.

Every stored page carries a checksum that is verified on every read. A page that fails is treated as absent and recomputed, so a damaged file costs performance and can never affect a response. The cache survives a restart.

What it costs

A hit copies the blocks back over PCIe, which is far cheaper than recomputing them but is not free. A miss costs nothing beyond the ordinary prefill that would have happened anyway, so the feature cannot make a request slower than not having it.

Memory is the real cost, and it is host memory rather than GPU memory. Watch that the tier plus everything else on the box still fits.

When it will not help

• Prompts that are never repeated. A cache with no reuse is pure overhead.
• Models whose prefix cache is disabled. Some hybrid-attention model families run with the prefix cache off for correctness, and tiering a disabled cache achieves nothing.
• Very short prompts. Re-computing 200 tokens is quicker than fetching them.

Related

• Memory & Sizing — how the GPU cache itself is sized.
• Load-Balancing Algorithms — KV-aware routing, which sends a request to the backend that already holds its prefix.