Troubleshooting & FAQ

Common questions and first checks when something is not behaving as expected in AgentWatch.

Do we have to change application code to adopt AgentWatch?

No. The lowest-friction path is changing one environment variable so an existing OpenAI or Anthropic SDK points at AgentWatch with an AgentWatch-issued key — no code changes at all. For estates too large even for that, transparent proxy mode routes existing applications through AgentWatch with no change whatsoever, not even the URL, by mapping provider hostnames to the gateway. Both paths keep full DLP, audit, and budget enforcement.

Does AgentWatch need GPUs?

No. AgentWatch is CPU-only. Model inference happens at the provider or on a customer-hosted inference cluster such as Lifeboat, vLLM, TGI, or Triton — AgentWatch is the governance and observability layer in front of it.

How do I check the gateway is reachable and which version is running?

curl -s https://<your-agentwatch-host>/health
# → {"ok":true,"version":"2.0.259", …}

If that answers and your request still fails, the problem is authentication, policy or routing — not connectivity. The next call to try is a minimal completion with your key, which exercises auth, the catalogue and the upstream in one go:

curl -s -X POST https://<your-agentwatch-host>/v1/chat/completions \
  -H "Authorization: Bearer ak_…" -H 'Content-Type: application/json' \
  -d '{"model":"<a model your org can use>","messages":[{"role":"user","content":"hi"}],"max_tokens":8}'

What do the common status codes mean?

Status	Meaning	First thing to check
401	Key not recognised, or an identity was required and not supplied	Is the key active in Employee Management? Was the employee removed (their key is revoked)? Is it the right organization?
402	A budget was exceeded and is being enforced	Which of org / team / user / key tripped — the strictest wins
403	Policy refused it: DLP block, model not available to the tenant, or an identity rejection	The code field names which
404	No route matched, or the model is not in the catalogue	Check the route’s path pattern and the model id
429	A rate limit — per tenant, per key, or the per-IP backstop	Rate-limit Alarms shows which
400 CONTEXT_LENGTH_EXCEEDED	The input does not fit the model’s context window	Shorten the input or use a larger-window model; this is terminal, not retryable
5xx	An upstream fault	Provider health and circuit-breaker state

I’m getting 402 responses. What does that mean?

A budget has been exceeded and the gateway is enforcing it rather than alerting. Two tiers enforce: the employee, always, and their team when Enforce team budgets is enabled. The message names the team when it was the team’s limit, so an employee who is inside their own budget is not sent to the wrong screen. Organization and per-key budgets are tracked and alerted on, not enforced at the gateway. Remedies are to raise the limit, grant a temporary override with an expiry, or wait for the daily or monthly window to reset.

I’m getting 429 responses. Which limit is it?

Rate limiting applies across many dimensions — model, user, team, organization, API key, application, MCP server, and individual tool — so check the scope in the audit record for the rejected request. AgentWatch deliberately returns 429 with a Retry-After header rather than a server error, so a well-behaved SDK backs off instead of retrying into the wall. The per-scope breach behavior is configurable: fall through to the next-priority provider, return 429, or alert.

DLP is blocking legitimate requests. How do I tune it?

Put the policy in log-only mode first and let it run against real traffic — that produces the false-positive set without disrupting anyone. Then raise the confidence threshold for the noisy category, adjust severity, or add allow-list entries. Detection confidence is configurable per category, and custom recognizers can be added per organization without code changes. Name detection in particular is tuned not to fire on ordinary factual questions.

A key is refused with “Model is not available to your team or tenant”

Check in this order:

1. Is the model in your organization’s catalogue? Settings → Model Catalog. A model belongs to one organization; a workspace that has never had it added will refuse it.
2. Has the team or employee been given an explicit model allowlist? An allowlist takes precedence over the organization default, so adding a model org-wide does not reach a team that has one.
3. Does the employee still exist? If they were removed, their key is revoked and now returns 401 Invalid API key.

Before 2.0.262 a third case produced this same message misleadingly: a removed employee’s key kept authenticating and every request was refused as though the team lacked model access. If you are on an older build and a key that “should work” reports this against a model its teammates can use, check whether that employee was deleted — and upgrade.

# what this key is actually allowed to use
curl -s https://<your-agentwatch-host>/v1/models \
  -H "Authorization: Bearer <the ak_ key>"

An empty list means the identity resolves but has no catalogue access; 401 means the key itself is not valid.

Requests are failing after a provider incident. What should I check?

Circuit-breaker state, visible in the metrics and the console. When a provider trips its breaker, AgentWatch fails over to the next-priority key or provider in the same model family — so confirm a fallback with the required capabilities is actually configured, and that its keys still authenticate. Per-key efficacy testing probes a key against the live provider to confirm it works and has quota.

/v1/responses returns 405 Method Not Allowed, but chat works. Why?

Before 2.0.375, the gateway built the upstream address for /v1/responses (and /v1/complete, /v1/generate, /v1/embed) by appending the request path to the provider’s Base URL. Most OpenAI-compatible providers are configured with a Base URL that already ends in /v1, for example https://llm.example.com/v1, so the request went to https://llm.example.com/v1/v1/responses. Many servers answer an unknown path like that with 405. Chat and embeddings were not affected, because they already handled the /v1 suffix.

Symptom	Cause	Fix
405 {"detail":"Method Not Allowed"} on /v1/responses, chat on the same model works	The provider’s Base URL ends in /v1 and the gateway is older than 2.0.375	Upgrade to 2.0.375 or later. No configuration change is needed.
Same error after upgrading	The provider does not serve the Responses API	Call it directly to confirm (below); if it also fails there, use /v1/chat/completions

To check the provider itself, bypassing the gateway:

curl -s -X POST https://llm.example.com/v1/responses \
  -H "Authorization: Bearer $PROVIDER_KEY" -H "Content-Type: application/json" \
  -d '{"model":"your-model","input":"say hi"}'

A JSON body with "object":"response" means the provider supports it, and the gateway will too from 2.0.375.

An agent’s requests are being refused with a policy message. Why?

That is insider-risk enforcement: the agent’s risk score crossed the configured block threshold, so the request was stopped before reaching the provider. The Enforcement view shows current actor standings, the active policy, and a decision log for every outcome. Enforcement is off by default and only applies once a policy is configured and enabled; thresholds for warn, step-up, and block are all adjustable, and policy changes take effect immediately.

The admin console slows down or stops loading while a Connect installer downloads. Is the server blocked?

No. This affects only a super administrator or tenant administrator who keeps the admin console (dashboard) open in the same browser that is downloading a Connect installer from the enrollment portal. Employees downloading from the portal are not affected.

It is a browser and HTTP/2 limitation, not a server problem. A browser sends every request to one host over a single HTTP/2 connection. While the 60-80 MB installer downloads, the dashboard’s requests wait behind installer data already queued on that connection, so the dashboard loads very slowly or seems not to load at all. The server keeps answering everyone else normally.

Measured on 2.0.372	Dashboard load
No download	about 8 s
Installer downloading in the same browser	30-38 s
Installer downloading on a separate connection	about 8 s
Server delivering the installer at 170-310 MB/s	about 8 s

Fix: open the admin console in a different browser (for example Edge while the download runs in Chrome). It gets its own connection and works normally. Or wait for the download to finish; the console recovers on its own. No setting or server change is needed.

Nothing appears in the dashboards. Is it broken?

Two common causes. First, check whether demo mode is on — it is enabled by default for new installations and shows generated sample data rather than yours; it is toggleable per user and can be disabled globally at install time. Second, if real traffic is expected but absent, confirm applications are actually pointed at the gateway and that their keys are valid, since a misconfigured client typically still reaches its provider directly.

I set a data retention policy but nothing is being deleted

Until 2.0.298 the three limits under Settings → Compliance → Data Retention Policies were recorded and never acted on: they saved successfully and no data was ever removed. If you set them expecting old records to age out, they did not.

From 2.0.298 deletion is real but opt-in. In the same panel, tick “Actually delete data older than these limits”. It is off on every deployment, including after this upgrade, because the numbers already stored were chosen while nothing enforced them — switching them on automatically would delete history nobody had reviewed, and deletion is permanent.

Symptom	Cause	Fix
Limits saved, nothing deleted	The enforcement switch is off (the default)	Tick Actually delete data older than these limits for that organization
One data type never shrinks	That limit is 0, which means do not delete — it is never read as “delete everything”	Set a positive number of days
A short limit is ignored	Anything under 7 days is refused	Use 7 or more
Deleted less than expected	Each run is batched and capped, so a first sweep over years of history takes several runs	Let the daily sweep run again; the log line retention-sweep-complete reports rows removed
No record of what was deleted	—	Each sweep writes one retention.swept entry per organization to Settings → Audit Logs with the exact counts

To see what would be removed without deleting anything, run the deployment with AW_RETENTION_DRY_RUN=1. AW_RETENTION_SWEEP=0 disables the sweep entirely.

Is my audit export complete?

Exports are capped at 100,000 rows, and a truncated export says so. Before 2.0.298 it did not — the file reported the surviving row count as the total, so an export covering a busy period looked complete while silently missing everything past the cap.

Check the JSON export for "complete": false; it also carries truncated, rowCap and truncatedSections naming which sections were cut. The CSV carries a banner beginning # ⚠ INCOMPLETE at the top of the file and again at the end of each truncated section.

If you see it, the file you are holding is a page of the record, not the record. Narrow the window and export again:

curl -s -o audit-30d.json \
  -H "Authorization: Bearer <admin JWT>" \
  "https://<your-agentwatch-host>/api/admin/audit/export?format=json&days=30"

Filtering by provider, source or tagKey/tagValue narrows it further. Operators can also watch for the audit-export-truncated log line, which fires whenever a truncated export is served.

SAML sign-in fails and the IdP gives no useful reason

On SAML the entity ID and ACS URL must match your IdP’s registration exactly; when they do not, the sign-in fails with nothing saying what differed. The two values your IdP needs from AgentWatch are:

Field	Value
Entity ID / Identifier	https://<your-agentwatch-host>/api/auth/saml/metadata/<provider id>
ACS / Reply URL	https://<your-agentwatch-host>/api/auth/saml/callback/<provider id>

Confirm what this deployment actually publishes, and register exactly that:

curl -s https://<your-agentwatch-host>/api/auth/saml/metadata/<provider id>

Note that the SAML Issuer (Entity ID) field on the provider is your IdP’s entity ID (for Entra, https://sts.windows.net/{tenant}/), not AgentWatch’s — see Single sign-on.

If the host in that metadata is not the address your users reach, set AW_PUBLIC_BASE_URL to your public URL so the published values do not depend on how the request arrived.

A setting I changed as super administrator did not reach my tenants

An organization that has its own settings row does not inherit platform defaults — and every active organization has one — so editing the platform row alone changes nothing for existing tenants. Push the specific fields instead: Settings → Company Settings → Apply platform defaults to all organizations, which previews exactly which tenants would change before anything is written. See Configuration and administration.

Do upgrades require a migration step?

No. Schema changes are applied and backfilled automatically at boot, so an upgrade is an image swap that keeps the data volume. Multi-instance deployments support zero-downtime rolling upgrades — drain a node, upgrade it, return it to the pool — with in-flight requests preserved through a graceful shutdown window. Documented rollback procedures and a pre-upgrade checklist accompany each release. Keep the data volume: do not remove volumes when bringing the stack down.

How is AgentWatch different from an AI observability platform?

Observability tools record what happened. AgentWatch enforces inline, before the request leaves: a budget breach rejects the request instead of alerting after the money is spent; DLP blocks or redacts before the provider sees the prompt rather than scrubbing a trace afterwards; prompt injection is blocked at the gateway rather than annotated. It also covers surfaces observability tools cannot see at all — unmanaged public AI usage on employee devices, MCP tool calls, and autonomous agents as their own risk class. AgentWatch also exports OpenTelemetry and Prometheus telemetry, so it complements an existing APM rather than replacing it.

Why did a large request return 503 GATEWAY_SCAN_TIMEOUT?

Screening a prompt is bounded in time. When a request is large enough that the model-backed scanners cannot finish within that budget, AgentWatch refuses the request with 503 GATEWAY_SCAN_TIMEOUT rather than forwarding it unscreened. The response names how long screening ran, the limit in force, and how far it got, and the condition is retryable — a smaller prompt, or one whose content has largely been screened before, will succeed.

This is deliberately a refusal and not a silent pass. Screening is a security control; abandoning it part-way and sending the request anyway would turn a latency problem into a policy gap.

Symptom	Cause	Fix
503 GATEWAY_SCAN_TIMEOUT on large prompts	Screening could not finish in the time allowed	Raise AW_GUARD_SCAN_BUDGET_MS, reduce the prompt, or disable model-backed scanners you do not need
Request dropped with no status and no body at a fixed time	An idle timeout in a proxy or load balancer in front of AgentWatch, not AgentWatch itself	Raise that timeout, and set AW_GATEWAY_TOTAL_BUDGET_MS below it (see next section)
Screening runs past the configured limit	Before 2.0.319 the limit applied per scanner rather than per request	Upgrade to 2.0.319 or later

Making the limit account for the whole request

AW_GUARD_SCAN_BUDGET_MS bounds screening only — the model call happens after it. A request that screens for 44 seconds and then calls the model can still exceed a 50-second proxy timeout, and the scan-timeout refusal will not fire because screening stayed inside its own budget.

If a proxy in front of AgentWatch closes idle connections, set the total budget below that timeout instead:

AW_GATEWAY_TOTAL_BUDGET_MS=45000     # keep the whole request under the proxy's limit
AW_GATEWAY_UPSTREAM_RESERVE_MS=15000 # time reserved for the model to answer

Screening then receives the remainder (30 s here) and refuses with a retryable 503 in time for the client to see it. Leaving AW_GATEWAY_TOTAL_BUDGET_MS unset keeps the previous behaviour.

How much CPU and memory does screening need?

The model-backed scanners (prompt injection, toxicity, bias, gibberish, ban-topics) run a transformer over the prompt in overlapping windows. That work is proportional to prompt size, and when a scanner is set to Block it sits in front of the request, so it is the figure to size against.

Measured on our own deployment, one blocking scanner, cold cache (the worst case — a prompt whose content has never been screened before):

Prompt	Blocking scan	Within the 45-second budget?
8k tokens	~7 s	yes
16k	~13 s	yes
32k	~27 s	yes
64k	~54 s	no — refused
262k	~220 s	no — refused

Adding CPU cores to a single AgentWatch process does not fix the larger sizes. Screening uses one inference runtime per process, and the threads inside it stop helping after about eight — measured, going from 8 threads to 12 improved a scan by 9%. A 64,000-token blocking scan is therefore roughly 54 seconds however large the host is.

Running screening in several processes

AW_GUARD_PROCESSES runs screening in that many child processes, each with its own inference runtime, and divides the windows between them. This is the setting that lifts the ceiling above.

AW_GUARD_PROCESSES=4      # default 1

Measured on a 120,000-character prompt: 1 process 45.6 s, 2 processes 29.7 s, 4 processes 23.8 s, with the number of model runs identical in every case — the same work, divided.

Budget roughly 2.5 GB of RAM per process, because each child loads its own copy of the models, plus a core or two each. Four processes is around 10 GB. The setting is capped at 8 (AW_GUARD_PROCESSES_MAX) so a large number cannot quietly exhaust a host’s memory.

Recommended starting points:

Workload	Setting	Host
Blocking, prompts under ~32k tokens	default (1)	9 cores, 8 GB
Blocking, prompts to ~64k	AW_GUARD_PROCESSES=2	16 cores, 12 GB
Blocking, prompts to ~262k	AW_GUARD_PROCESSES=8	64 cores, 24 GB
Log only, any size	default (1)	screening is off the critical path

DLP is not affected by any of this. Pattern-based DLP (credit cards, SSNs, secrets) adds no measurable latency — measured between −2 s and +3 s on prompts from 16k to 128k tokens, which is indistinguishable from noise. The costs on this page are the model-backed scanners only.

If you need blocking screening on very large prompts with low latency, the practical options today are to run the scanners in Log only for that traffic and block on a narrower set of content, or to raise AW_GUARD_PROCESSES and give the host the memory to match.

Why did a request return 400 or 503 saying the model has no provider?

503 CONFIG_UNAVAILABLE means the model catalogue could not be loaded at that moment — a transient backend condition, not a statement that the model is unconfigured. It is safe and correct to retry.

Before 2.0.319 this case produced 400 NO_PROVIDER_FOR_MODEL, a configuration error that client SDKs treat as permanent and never retry, so a brief database hiccup surfaced as a dead request. If you still see the 400 intermittently for a model that works seconds either side, upgrade.

A 400 NO_PROVIDER_FOR_MODEL that happens consistently for one model is the real thing: that model has no provider configured for the organization. Check Settings → Model Catalog.

Does every scanner read my whole prompt?

Not by default. prompt-injection and toxicity window the entire prompt. ban-topics, bias and gibberish read a leading excerpt of roughly 2,000 characters and judge that, because windowing them makes their cost grow with prompt size.

Every scanner result now reports what it actually inspected, so a verdict covering an excerpt is visibly different from one covering the whole prompt. To screen the full prompt with those scanners, opt in and accept the additional scan time:

AW_GUARD_FULL_TEXT_SCANNERS=ban-topics,bias,gibberish

Where do I get help?

Contact your Iterate.ai representative with your AgentWatch version, deployment mode (self-hosted, SaaS, hybrid, or air-gapped), the health endpoint output, and the audit record or trace ID for the affected request. Diagnostics endpoints are available for support investigations.

Sources and references

• Enforcement behavior in depth: Governance and cost control.
• Deployment and sizing: Deployment.