Sizing and capacity planning

Every number on this page was measured on AgentWatch itself rather than estimated. Where a figure is derived from a measured constant rather than observed directly, it says so.

What actually drives the cost?

Three things, in order of importance. Getting these right matters more than any other sizing decision.

1. Which engines you enable. AgentWatch has two independent screening engines and they differ by three orders of magnitude:

Engine	What it does	Cost
DLP	PII, secrets, card numbers — pattern matching	Negligible. 3.5 ms for a 30k-token prompt, 77 ms for 300k, 670 ms for 1M
Guardrails	Prompt injection, toxicity, bias, gibberish, banned topics — transformer models	Seconds. 0.06–1.24 s per 1,000 tokens per scanner

If you run DLP only, AgentWatch is effectively free and your capacity is bounded by your LLM provider, not by the gateway. Everything else on this page is about the guardrail scanners.

2. New content, not context size. Screening results are cached per content window, so a prompt is only screened for the part that has not been seen before. Measured with a 262,000-token context:

Turn	Time	Model runs
First (nothing cached)	152 s	480
Same context again	15 ms	0
Context + 4k new tokens	2.84 s	9
Context + 4k more	2.61 s	10

A 262k-token conversation and a 36k-token one cost the same per turn once running — 9 to 10 model runs. Context size only matters for the first turn.

3. Blocking versus log-only. Log-only screening runs beside the request and adds no latency. Blocking screening has to finish before the answer is released. See Latency.

How much does DLP cost?

Measured directly rather than end to end, because against a real model call the provider’s own variance (65–213 s on large prompts) is far larger than anything DLP does:

Prompt	DLP pattern pass
100k chars (~30k tokens)	3.5 ms
1M chars (~300k tokens)	77 ms
3.4M chars (~1M tokens)	670 ms

DLP is not a capacity consideration at any realistic size. If you run DLP only, size for your LLM traffic and ignore the rest of this page.

Since 2.0.335 the pattern pass runs in worker threads rather than on the thread that serves requests, and scans the messages of a prompt in parallel. Two things improve as a result, both measured:

	Before	After
Longest pause in serving other requests (1M-token prompt)	829 ms	1 ms
Scanning an 8-message prompt	210 ms	86 ms

The first row is the one that matters on a shared gateway: previously a single large prompt stopped every other request for the duration of its scan. Findings are identical either way — the work moved, it did not change.

What does each scanner cost?

Measured on a 60,000-character prompt (≈18,000 tokens), 8 inference threads, one process, nothing cached:

Scanner	Time	Per 1k tokens	Model on disk
Banned topics	22.3 s	1.24 s	728 MB
Prompt injection	14.1 s	0.78 s	948 MB
Toxicity	3.2 s	0.18 s	433 MB
Gibberish	1.4 s	0.08 s	256 MB
Bias	1.1 s	0.06 s	shared
Secret detection	2 ms	~0	none (pattern-based)
Invisible text	3 ms	~0	none (pattern-based)
All five model scanners	42.1 s	2.34 s	2.4 GB

Banned topics is the most expensive scanner, ahead of prompt injection. Enabling all five costs about three times prompt injection alone. Size for the set you actually enable — the difference between two scanners and five is a factor of three in hardware.

In CPU-seconds (the unit capacity planning needs):

All five scanners: ~18.7 CPU-seconds per 1,000 new tokens
Prompt injection only: ~6.3 CPU-seconds per 1,000 new tokens
DLP only: negligible

How fast is a cold prompt on different hardware?

A cold prompt is text AgentWatch has never screened before, so every window runs through the guardrail models. It is the worst case: in a conversation only the new message is cold, and the rest comes from the screening cache (about 1 second for an 8,000-token prompt seen before, whatever the hardware).

Measured

8,000-token cold prompt, screening only (no upstream model call), verdict cache off, fresh text on every run. Measured October 2026 through the gateway’s own large-prompt screening path.

Machine	Screening threads	Default scanners¹	All five model scanners²
AWS `t3.xlarge` (4 vCPU burstable, 16 GB)	2 (automatic)	19.6 s	—
Apple M4 Max (16 cores, 64 GB), limited to 2 threads	2	18.3 s	—
Apple M4 Max, limited to 4 threads	4	10.2 s	—
Apple M4 Max, automatic	8	5.9 s	10.7 s
Apple M4 Max, 12 threads	12	4.9–5.8 s	—

¹ Prompt injection, toxicity and gibberish, plus the pattern-based secret and invisible-text checks — what is on when you enable guardrails without changing the scanner list. ² Adds banned topics and bias.

Two things the table shows:

Threads are the main lever, and they size themselves. Each screening process uses one thread per available CPU minus one, capped at 8. Beyond 8 the gain is small (12 threads measured only 0–17% faster), which is why the cap is there.
Two threads reproduce the 4-vCPU cloud box almost exactly (18.3 s vs 19.6 s), so the thread rows are a fair guide to what a server with that many usable threads will do.

Expected time by CPU and memory

Projected from the measurements above. Cloud vCPUs are usually hyperthreads — 16 vCPU is 8 physical cores — and server cores are slower per core than the M4 Max’s, so cloud figures are given as ranges. Treat them as planning numbers and measure on your own hardware before committing.

Box	Threads (automatic)	Cold 8k, default scanners	Cold 8k, all five	Memory to plan for
4 vCPU / 16 GB	2–3	15–20 s	30–40 s	fits one process (6–8 GB used)
8 vCPU / 16 GB	7	9–13 s	17–24 s	fits one process with all five
16 vCPU / 32 GB	8	6–10 s	11–18 s	room for 2–4 processes (default) or 2–3 (all five)
32 vCPU / 64 GB	8 per process	5–7 s	9–13 s	room for 4+ processes, for throughput

A prompt’s own screening does not get faster beyond 8 threads. A bigger box buys throughput — more prompts screened at the same time — through extra screening processes, not a faster single prompt.

When do extra screening processes help?

AW_GUARD_PROCESSES runs screening in several processes, each with its own copy of the models and its share of the threads. It is off by default (one process).

Measured peak memory of one screening process (models loaded, screening an 8,000-token prompt): 3.6 GB with the default scanners, 5.8 GB with all five. That is more than the models’ size on disk, because the runtime holds working buffers alongside them.

Cold prompt (default scanners, M4 Max)	1 process	2 processes	3 processes
8,000 tokens	5.9 s	6.5 s	5.9 s
32,000 tokens	27.2 s	24.2 s	21.2 s

For prompts of a few thousand tokens, extra processes do not help — the work is too small to be worth splitting, and each process gets fewer threads.
For very large prompts (tens of thousands of tokens) they help modestly: three processes screened a 32,000-token prompt 22% faster.
For many simultaneous users they add parallel capacity. Plan memory as roughly 2 GB for AgentWatch itself plus 3.6 GB (default scanners) or 5.8 GB (all five) per screening process.

Recommendations

Use a compute-optimized, non-burstable instance (for example AWS c7i, Azure F-series). Screening is sustained CPU work, which is exactly what burstable instances throttle once their credits run out.
Leave threads and processes on automatic unless your traffic is mostly very large prompts; then try AW_GUARD_PROCESSES=2 or 3 and measure.
Enable banned topics and bias only if you need them — they nearly double cold screening time.
16 vCPU / 32 GB is a sound default for a production deployment with guardrails on: 6–10 s for a cold 8,000-token prompt, about 1 s once cached, and memory for two to four screening processes if traffic needs them.

How many users will a given box support?

Capacity is throughput, so it depends on how often each participant sends a turn. The two profiles behave very differently:

A person sends a turn every ~30 seconds (they read and think between turns)
An agent in a loop sends one every ~15 seconds, continuously

Assuming a typical turn carries 4,000 new tokens (a tool result or a reasonable message), derived from the measured constants above:

DLP only

Concurrent users	CPU	Memory
5 – 200	2–4 cores	4 GB

Pattern matching does not scale meaningfully with user count at these volumes. Size for your LLM traffic, not for AgentWatch.

Prompt injection blocking (one scanner)

Concurrent	People	Agents
5	2 cores, 8 GB	9 cores, 12 GB
20	3 cores, 8 GB	35 cores, 24 GB
50	5 cores, 12 GB	85 cores, 48 GB
100	9 cores, 16 GB	170 cores, 64 GB
150	13 cores, 16 GB	255 cores, 96 GB
200	17 cores, 24 GB	340 cores, 128 GB

All five scanners blocking

Concurrent	People	Agents
5	3 cores, 12 GB	26 cores, 24 GB
20	6 cores, 16 GB	101 cores, 48 GB
50	12 cores, 24 GB	251 cores, 96 GB
100	22 cores, 32 GB	not practical on CPU
150	32 cores, 48 GB	not practical on CPU
200	42 cores, 64 GB	not practical on CPU

Agents are 2–6× more expensive than people for the same headcount, because they do not pause. If your workload is agentic and you want all five scanners blocking, the honest advice is to run the expensive scanners in log-only and block on the one or two that matter for your risk model.

Memory is governed by screening processes, not by user count: roughly 2.5 GB per process with one scanner, 4 GB with all five, plus about 1 GB for the gateway itself — and, with Advanced NER enabled, roughly another 0.5–0.8 GB for its language model (an expected figure, not a measurement; see Disk). Set AW_GUARD_PROCESSES to about one quarter of your core count.

What about the first turn on a very large context?

The first turn of a conversation has nothing cached, so the whole context is screened. Derived from the measured per-token constants:

Context	All five scanners	Prompt injection only	Fits the 45 s budget?
64k	1,197 CPU-s	403 CPU-s	needs 27 / 9 cores
128k	2,394 CPU-s	806 CPU-s	needs 53 / 18 cores
262k	4,899 CPU-s	1,651 CPU-s	needs 109 / 37 cores
1M	18,700 CPU-s	6,300 CPU-s	needs 415 / 140 cores

Beyond about 128k tokens, screening an entire cold context with blocking model scanners stops being a CPU workload. Past the 45-second screening budget the request is refused with 503 GATEWAY_SCAN_TIMEOUT rather than forwarded unscreened.

For large contexts, use one of these instead:

DLP blocking plus guardrails in log-only. Data protection stays inline and costs nothing; injection and toxicity findings are still recorded, audited and visible — they simply do not hold the request.
AW_GUARD_PROCESSES to divide the first turn across child processes. Measured on a 120,000-character prompt: 1 process 45.6 s, 2 processes 29.7 s, 4 processes 23.8 s, with the model-run count identical in every case.
Accept the first-turn cost where contexts are long-lived, since every later turn is ~2.7 s regardless of context size.

How much latency does AgentWatch add?

Measured end to end against a live deployment, DLP in each mode, versus the same request with AgentWatch screening disabled:

Prompt	Log only	Redact	Block
16k tokens	−0.8 s	−0.8 s	−0.8 s
32k	+0.5 s	+0.3 s	+0.3 s
64k	+0.1 s	+1.1 s	+1.5 s
128k	−0.1 s	+0.0 s	+3.2 s

Negative figures are measurement noise against the variability of the model call itself. For DLP, all three modes are within noise up to 128k tokens.

For the guardrail scanners the picture depends entirely on posture:

Posture	Added latency
Disabled	none
Log only	none — screening runs beside the request, not in front of it
Blocking, standard	the full screening time (0.06–1.24 s per 1k new tokens per scanner)
Blocking, concurrent	screening, less whatever the provider call overlaps it — see below

Concurrent enforcement

Setting AW_GUARDRAILS_CONCURRENT=1 runs enforcing scans beside the provider call instead of in front of it, and withholds the response until the verdict arrives. Nothing is screened less — the same scanners, thresholds and block decisions apply — but the caller waits for whichever of the two takes longer rather than for both in sequence.

How much this saves depends entirely on your provider’s response time, and it is capped by it: the saving is roughly the time the provider takes, up to the time screening takes.

Measured on a live deployment, a 4,000-token prompt with a 3,000-token answer against a provider that takes about 44 seconds to generate it:

	Median
Inline (screening in front of the provider call)	54.2 s
Concurrent (screening beside it)	42.6 s

About 12 seconds saved, roughly 21% — and 42.6 s is essentially the provider’s own time (44.3 s measured directly against it), so screening disappeared behind the model call entirely. A refused prompt was still refused in the same run.

⚠ Against a fast provider there is nothing to hide behind. On a deployment whose provider answers in about 2 seconds while screening takes 20, we measured no difference at all. The saving is the overlap, so it is large when the model is the slower half and nil when it is not. Measure it on your own traffic.

⚠ A refused request becomes slower, not faster. It now waits for the provider call before the refusal is returned. Refusals are rare, and the trade is deliberate — the common case is a request that passes.

AW_GUARDRAILS_CONCURRENT=1    # default off

⚠ What this trades. The prompt reaches your provider before the guardrail verdict exists. For the model-backed scanners that is usually acceptable: their purpose is to stop you acting on injected or toxic content, and withholding the response preserves that. If your provider logs prompts and you would rather an injection attempt never reach it, leave this off and pay the latency.

DLP is never run this way. Its purpose is to stop sensitive data leaving at all, so it always completes before the request is forwarded. It is also cheap enough that it does not need this.

⚠ This changes latency, not capacity. The CPU work is identical, so the tables above are unchanged. Concurrent enforcement hides the wait; it does not reduce the cost.

Disk

What	Size	Notes
Container image	~3.5 GB	includes the Python NER components; ~3 GB before 2.0.363, plus the ~560 MB NER language model (an expected figure — see below)
Scanner models	2.4 GB	downloaded on first use; only if model-backed scanners are enabled
Verdict cache	up to ~3 MB	capped at 20,000 entries per process

The Advanced NER language model (2.0.363)

From 2.0.363 the image ships spaCy’s en_core_web_lg model for Advanced NER instead of en_core_web_sm, because the small model missed or mislabelled enough names, organizations and dates to produce wrong redactions.

	`en_core_web_sm` (before)	`en_core_web_lg` (2.0.363)
On disk (image size)	~12 MB	~560 MB
Extra memory while Advanced NER runs	negligible	roughly 0.5–0.8 GB

⚠ These two figures are expected values, not measurements. The disk figure is the model’s published size. The memory figure is derived from that size — almost all of it is the word-vector table, which spaCy loads into memory — and has not been measured on a running AgentWatch. Plan for the upper end. It is only paid when Advanced NER is enabled: the model is loaded by the NER process, which does not start otherwise.

To keep the smaller model, build the image with --build-arg SPACY_MODEL=en_core_web_sm, or set SPACY_MODEL=en_core_web_sm in the container environment of an image that has it installed. If the named model is not installed, AgentWatch falls back to whichever English model is, largest first, rather than running without NER.

Growth depends on traffic and retention. The event archive keeps 90 days by default, gzipped and rotated hourly; usage records are kept in the database until you prune them.

Start with 50 GB for a deployment of up to 50 users and grow from there; the dominant term is the event archive, which is proportional to the number of requests rather than to context size.

If you enable session vectors, each stored conversation adds a 384-dimension float vector (about 1.5 KB) plus its metadata.

Worked examples

A 20-person engineering team, DLP only, contexts up to 128k. 4 cores, 8 GB, 50 GB disk. AgentWatch adds no measurable latency.

The same team, plus prompt-injection blocking. 4 cores, 8 GB. Allow a few seconds on the first turn of a long conversation; later turns are unaffected.

Five production agents, all five scanners, 262k contexts. Either 26 cores and 24 GB with AW_GUARD_PROCESSES=6, or — the configuration we would recommend — 8 cores with DLP blocking, guardrails log-only and AW_GUARDRAILS_CONCURRENT=1, which keeps enforcement on the controls that matter without paying for it on every turn.

200 people, DLP blocking, guardrails log-only. 8 cores, 16 GB. Log-only screening consumes CPU but never delays a request, so size for throughput and not for latency.