Observability & Insights

AgentWatch’s observability layer is built on the fact that every request already passes through it. Nothing needs to be instrumented in application code to be measured.

What is captured per request?

Tokens (prompt, completion, and cached), cost, duration, time-to-first-token, model, provider, team, user, session, endpoint, status, and trace ID — plus an automatic task category (Code, Content, Data Analysis, or a custom category). Full conversation transcripts are retained, subject to DLP redaction policy, and are browsable per employee, per team, and per session — which answers “what did this person actually ask the model this morning?” without log archaeology.

What does the Observe workspace show?

A unified observability workspace inside the console, with themed dashboards across nine surfaces:

Surface	What it covers
AI	Request, token, cost, and latency KPIs; provider explorer; prompt lineage (DLP-scrubbed); agent topology; quality evaluations
Services	Distributed-trace waterfall and end-to-end service flow through the gateway pipeline
Fleet	Deployed-agent health grid, agent inventory, and version distribution
Optimize	Direct-versus-optimized comparison of model calls, tokens sent, cache hits, and estimated cost saved
Business	Request-to-outcome funnel and spend by team
Security	Risk-ranked DLP, firewall, and guardrail threats; attack mix; incident timeline
Insider Risk	AI-agent risk scoring and enforcement standings
Delivery	Release verification pass/warn/fail results
Logs	Query bar, volume histogram, and facets over gateway events

Can I query my AI usage in plain English?

Yes. AgentWatch includes a built-in analytics interface that answers natural-language questions over your own audit and usage data — for example “which teams’ usage grew more than 50% last month?”, “show every employee who triggered a DLP violation in the last 30 days by category”, or “which application caused this morning’s spike in rate-limit errors?”. Conversations persist so a thread can be resumed, shared with a colleague, and audited. Trend and anomaly detection flag deviations from baseline automatically. It is designed for compliance officers, FinOps analysts, and managers rather than for engineers writing queries, and it can be configured to run against a local model so the analysis itself stays in-boundary.

Which monitoring tools does it integrate with?

AgentWatch emits standard telemetry rather than requiring its own stack:

• OpenTelemetry (OTLP, Jaeger, Zipkin) for traces, metrics, and logs — consumed by Jaeger, Grafana Tempo, Datadog, Honeycomb, New Relic, or AWS X-Ray. Traces are mandatory for every tool call; there is no code path that skips them.
• Prometheus metrics at a standard endpoint — latency histograms, token counts per model, success and error rates, queue depths, circuit-breaker state.
• Langfuse for LLM-specific trace and generation tracking, with configurable sampling.
• Log platforms — structured JSON per request, ingested by ELK, Splunk, Loki, or Datadog Logs. Authentication failures, rate-limit breaches, DLP violations, and budget thresholds emit to separate streams, so a SIEM can subscribe selectively.
• Distributed tracing end to end across multi-step agent and tool workflows, with trace IDs propagating through gateway, provider, tool calls, and response synthesis.

How does the event pipeline work at scale?

An event stream covers request and response, DLP scans and violations, authentication outcomes, budget thresholds and breaches, provider health changes, rate limiting, agent heartbeats and captured traffic, MCP tool calls, guardrail and schema violations, and webhook deliveries. Each class is separated so security, FinOps, and SRE teams subscribe only to what they need. Optional Kafka and a columnar analytics database support high-volume deployments where dashboards must slice tens of millions of rows by any dimension in sub-second time; a file-based archiver covers smaller deployments. All of it runs inside the customer environment — no event leaves the boundary unless an outbound webhook or exporter is explicitly configured.

What about audit and reporting?

Audit logs are retained per configurable policy (default one year for audit records) and export as JSON, CSV, or PDF from the UI or programmatically for SIEM ingestion. Access to audit data is itself permissioned, and the permission to read audit records is separable from the permission to export them — so an analyst can investigate without being able to exfiltrate.

Where do I look first?

Usage Insights is the per-request record: who called, which model, tokens, cost, latency, DLP outcome, and the full transcript on drill-down. Observe is the aggregated workspace across usage, security, fleet, and delivery.

Filter the audit trail by employee, model, status or custom tag, then open a row to see the conversation exactly as it was sent and returned.

How do I get the data out?

# the audit trail, filtered
curl -s "https://<your-agentwatch-host>/admin/audit/trail?limit=100" \
  -H "Authorization: Bearer <tenant-admin JWT>"

# a compliance export
curl -s "https://<your-agentwatch-host>/api/admin/audit/export?from=2026-09-01&to=2026-09-30" \
  -H "Authorization: Bearer <tenant-admin JWT>" -o audit-september.json

The audit trail pages by conversation, not by row. limit (at most 1,000) and total count conversations, and each conversation on the page comes with all of its turns, so data can hold more rows than limit. The response says so itself: pageUnit is "conversation", conversationCount is the number of conversations on the page, and rowCount the number of rows. To fetch the next page, add limit to offset.

The Observe workspace reads the same numbers through a documented API, so anything on a dashboard can go into your own reporting:

curl -s "https://<your-agentwatch-host>/admin/api/observability/kpis?window=7d" \
  -H "Authorization: Bearer <tenant-admin JWT>"
curl -s "https://<your-agentwatch-host>/admin/api/observability/timeseries/cost?groupBy=team&window=30d" \
  -H "Authorization: Bearer <tenant-admin JWT>"
curl -s "https://<your-agentwatch-host>/admin/api/observability/model-usage?window=30d" \
  -H "Authorization: Bearer <tenant-admin JWT>"

What is in each column of the CSV export?

The CSV is written as UTF-8 with a byte-order mark and served as text/csv; charset=utf-8. The BOM is what stops Excel opening the file in the system codepage — without it, em-dashes and other non-ASCII characters arrive corrupted even though the bytes were always correct.

Column	What it is
Timestamp	When the request was recorded
Employee	The identity the request was attributed to
LLM Account / Verified / Plan	The consumer AI account behind a request — a signed-in ChatGPT or Claude session captured from a browser or an enrolled endpoint. Traffic authenticated with a gateway API key has no such account, so these are empty by design on an API-only deployment
Model	The model that was requested
Task Type (best-effort)	A keyword heuristic over the prompt text. Labelled best-effort because it is one — do not build reporting on it; use Tool Calls and Turn instead
Tool Calls	How many tool calls the response made. A fact, recorded at write time; populates for every request the gateway proxies
Turn	user when a person’s message drove the turn, agent when the model was being handed its own tool’s output. This is how you separate one human request from the forty internal steps of an agent run
Query Preview / Response Preview	The first AW_USAGE_PREVIEW_CHARS characters (default 1000). Tool calls appear with their arguments, and an agent turn shows the tool result it was reacting to
Tokens / Duration / Status	As recorded
Cost	Blank when the model has no pricing configured — see below

Why is the Cost column blank or zero?

A blank cost and a 0.0000 cost mean different things, deliberately:

You see	It means	What to do
A number	The model is priced and this is the computed cost	—
0.0000	The model is priced, and this call genuinely cost that	—
Blank	No pricing exists for this model — nothing was measured	Set the rates (below)

Self-hosted and newly imported models arrive with no rates, so every call against them costs nothing as far as the gateway knows. Earlier releases printed 0.0000 in all three cases, which a finance report then summed as though it had been measured. Set the rates once and every later row is costed:

Console path: Settings → Models → (your model) → Edit → Cost per 1M tokens.

curl -s -X PUT "https://<your-agentwatch-host>/admin/models/<model-id>" \
  -H "Authorization: Bearer <admin JWT>" -H 'Content-Type: application/json' \
  -d '{"inputCostPer1m": 0.35, "outputCostPer1m": 1.40}'

You should now see a cost on new rows for that model. Pricing is not retroactive — rows already written keep the cost they were recorded with.

How do I see what an agent actually did?

Raise the preview budget and the tool-call arguments come with it. Set AW_USAGE_PREVIEW_CHARS on the gateway (default 1000, clamped to 200–20000) and restart:

AW_USAGE_PREVIEW_CHARS=4000

Response previews then carry [Tool Calls: search_docs({"q":"refund policy"})] alongside the model’s prose rather than instead of it, and a turn that fed a tool result back shows [Tool Results: toolu_01 → …] in place of the old [non-text content] placeholder. Filter the export on Turn = agent to see only the internal steps.

The cap applies after DLP scrubbing, so raising it never widens what a redacted row exposes.

How do I wire it into our monitoring?

Prometheus scrapes the gateway directly:

curl -s https://<your-agentwatch-host>/metrics | head

OpenTelemetry traces export to any OTLP collector, and the event pipeline streams every request to Kafka or ClickHouse for long-term analytics. Configure both under Settings → System; the panel shows live pipeline status rather than a static label, so you can tell a running pipeline from a configured one.

For alerting, webhooks fire on security and budget events — including security.rate_limit_exceeded and security.suspicious_activity.

The dashboards are empty. Is it broken?

Usually not — it means what it says. Work down this list:

Check	Why
Is demo mode off?	Outside demo mode the console shows real data only; a quiet deployment shows empty panels by design
Are you signed in with an admin role?	The observability API requires super-admin or admin; an unauthenticated probe returns nothing and the panel reports “no data”
Has any traffic gone through the gateway in the window?	Widen the window, or send one test request
Are you looking at the right organization?	A super-admin with no ?organizationId= sees the platform scope

Panels state which of live, demo data or no data they are showing, so a populated-looking dashboard is never fabricated — if it says live, a query returned those rows.

Why are my audio and image calls missing from a per-organization read of the archive?

They were being written without an organization, so they landed in a separate directory (events/_platform/…) from chat events (events/platform_<org-id>/…). The records were always produced — reading the archive by organization simply did not find them. Fixed in 2.0.295; events written after upgrading carry the organization.

Archive files already written keep the path they were written to. If you need a complete per-organization history across the upgrade, read both locations for that period.

Why was a blocked image or audio request charged?

It should not have been, and is not from 2.0.295. A request refused by policy never reaches a provider, so it costs nothing — a blocked chat request has always recorded zero. Blocked image and audio requests were being priced on their prompt tokens anyway. Audio made it plainer: it is billed per minute, and a refused upload has no duration at all.

Rows already written keep the cost they were recorded with.

Sources and references

• Compliance retention settings: Security and compliance.
• Cost reporting and chargeback: Governance and cost control.