Observability & Insights
AgentWatch’s observability layer is built on the fact that every request already passes through it. Nothing needs to be instrumented in application code to be measured.
What is captured per request?
Section titled “What is captured per request?”Tokens (prompt, completion, and cached), cost, duration, time-to-first-token, model, provider, team, user, session, endpoint, status, and trace ID — plus an automatic task category (Code, Content, Data Analysis, or a custom category). Full conversation transcripts are retained, subject to DLP redaction policy, and are browsable per employee, per team, and per session — which answers “what did this person actually ask the model this morning?” without log archaeology.
What does the Observe workspace show?
Section titled “What does the Observe workspace show?”A unified observability workspace inside the console, with themed dashboards across nine surfaces:
| Surface | What it covers |
|---|---|
| AI | Request, token, cost, and latency KPIs; provider explorer; prompt lineage (DLP-scrubbed); agent topology; quality evaluations |
| Services | Distributed-trace waterfall and end-to-end service flow through the gateway pipeline |
| Fleet | Deployed-agent health grid, agent inventory, and version distribution |
| Optimize | Direct-versus-optimized comparison of model calls, tokens sent, cache hits, and estimated cost saved |
| Business | Request-to-outcome funnel and spend by team |
| Security | Risk-ranked DLP, firewall, and guardrail threats; attack mix; incident timeline |
| Insider Risk | AI-agent risk scoring and enforcement standings |
| Delivery | Release verification pass/warn/fail results |
| Logs | Query bar, volume histogram, and facets over gateway events |
Can I query my AI usage in plain English?
Section titled “Can I query my AI usage in plain English?”Yes. AgentWatch includes a built-in analytics interface that answers natural-language questions over your own audit and usage data — for example “which teams’ usage grew more than 50% last month?”, “show every employee who triggered a DLP violation in the last 30 days by category”, or “which application caused this morning’s spike in rate-limit errors?”. Conversations persist so a thread can be resumed, shared with a colleague, and audited. Trend and anomaly detection flag deviations from baseline automatically. It is designed for compliance officers, FinOps analysts, and managers rather than for engineers writing queries, and it can be configured to run against a local model so the analysis itself stays in-boundary.
Which monitoring tools does it integrate with?
Section titled “Which monitoring tools does it integrate with?”AgentWatch emits standard telemetry rather than requiring its own stack:
- OpenTelemetry (OTLP, Jaeger, Zipkin) for traces, metrics, and logs — consumed by Jaeger, Grafana Tempo, Datadog, Honeycomb, New Relic, or AWS X-Ray. Traces are mandatory for every tool call; there is no code path that skips them.
- Prometheus metrics at a standard endpoint — latency histograms, token counts per model, success and error rates, queue depths, circuit-breaker state.
- Langfuse for LLM-specific trace and generation tracking, with configurable sampling.
- Log platforms — structured JSON per request, ingested by ELK, Splunk, Loki, or Datadog Logs. Authentication failures, rate-limit breaches, DLP violations, and budget thresholds emit to separate streams, so a SIEM can subscribe selectively.
- Distributed tracing end to end across multi-step agent and tool workflows, with trace IDs propagating through gateway, provider, tool calls, and response synthesis.
How does the event pipeline work at scale?
Section titled “How does the event pipeline work at scale?”An event stream covers request and response, DLP scans and violations, authentication outcomes, budget thresholds and breaches, provider health changes, rate limiting, agent heartbeats and captured traffic, MCP tool calls, guardrail and schema violations, and webhook deliveries. Each class is separated so security, FinOps, and SRE teams subscribe only to what they need. Optional Kafka and a columnar analytics database support high-volume deployments where dashboards must slice tens of millions of rows by any dimension in sub-second time; a file-based archiver covers smaller deployments. All of it runs inside the customer environment — no event leaves the boundary unless an outbound webhook or exporter is explicitly configured.
What about audit and reporting?
Section titled “What about audit and reporting?”Audit logs are retained per configurable policy (default one year for audit records) and export as JSON, CSV, or PDF from the UI or programmatically for SIEM ingestion. Access to audit data is itself permissioned, and the permission to read audit records is separable from the permission to export them — so an analyst can investigate without being able to exfiltrate.
Sources and references
Section titled “Sources and references”- Compliance retention settings: Security and compliance.
- Cost reporting and chargeback: Governance and cost control.