Skip to content
Docs

Observability and Telemetry

OpenTelemetry: OTLP (default), Jaeger, Zipkin, Console exporters. Traces/metrics/logs. Mandatory OTel traces are emitted for every tool call (MCP + native tool invocations) — there is no code path that bypasses tracing. Any OTLP-compatible APM ingests the stream:

  • Tracing: Jaeger, Grafana Tempo, Datadog APM, Honeycomb, New Relic, AWS X-Ray
  • Metrics: Prometheus (scrape /metrics), Datadog, Grafana Cloud, New Relic
  • Logs: ELK (Elasticsearch + Logstash + Kibana), Splunk, Grafana Loki, Datadog Logs

Langfuse: Trace logging, generation tracking, session/user tracking, configurable sample rates. Prometheus: GET /metrics — counters, gauges, histograms, system metrics (latency histograms, token counts per model, success/error rates, queue depths, circuit-breaker state). Usage: Per-request tokens, cost, duration, TTFT, sessionId, model, provider, team, user — filterable/aggregatable by any metadata tag (Section 22 events). Real-time + historical with configurable alert thresholds. Distributed tracing: end-to-end across multi-step agent + MCP + pipeline workflows; trace IDs propagate through gateway → upstream provider → tool calls → response synthesis, with each hop captured as a span. Structured logging: every request emits JSON with trace_id, user_id, team_id, organization_id, endpoint, model, provider, status, latency_ms, prompt_tokens, completion_tokens, cost_usd. Raw prompts can be elided to redacted summaries via DLP policy. Separate log streams: authentication failures (AuthEvent), rate-limit breaches, DLP violations, and budget thresholds emit to dedicated channels so a SIEM can subscribe selectively. Audit: Flat-file ndjson.gz per org, CSV / JSON / PDF export, LLM-powered analysis (Admin Insights), SIEM export via Kafka or webhooks. Full conversation history: every request is captured with full prompt + response transcript (DLP-redacted per policy), browsable per employee, per team, or per session — answers “what did this user actually ask Claude this morning?” without needing to scrape logs. AI task auto-categorization: every LLM request is classified at the gateway into broad task categories — Code, Content, Data Analysis, plus extensible custom categories — so usage reports can show “60% of last quarter’s GPT-4 spend went to Code Generation, 25% to Content, 15% to Data Analysis.” Categorization is metadata only, captured alongside tokens and cost in the same event. Performance graphs + analytics leaderboards: out-of-the-box dashboards rank teams and employees by spend, request volume, tokens-per-dollar efficiency, and month-over-month growth — useful for FinOps reviews and for identifying internal champions or runaway costs early.

21.1 AI Insights — Natural-Language Query over Audit Data

Section titled “21.1 AI Insights — Natural-Language Query over Audit Data”

AgentWatch ships a built-in LLM-powered analytics interface that lets non-technical administrators ask questions of the audit + usage data in plain English. Conversations persist in the insights_conversations table so the analyst can resume a thread, share it with a teammate, and audit who-asked-what (multi-turn natural-language analytics with thread history). Ten endpoints under /api/admin/insights/… cover chat (streaming + non-streaming), conversation list/get/delete, and initialization. Backed by a RAG pipeline over the conversation/audit/event store, it answers questions like:

  • “Which teams’ Claude usage grew more than 50% last month?”
  • “Show me every employee who triggered a DLP violation in the last 30 days, grouped by category.”
  • “What’s the highest-cost prompt my organization sent yesterday and which model handled it?”
  • “Which application is responsible for the spike in 429 responses this morning?”

Configurable to use any approved upstream model (Ollama first for sovereign deployments, Anthropic or OpenAI for SaaS). Trend detection and anomaly identification flag deviations from baseline automatically. The interface is designed for compliance officers, FinOps analysts, and engineering managers — not for engineers writing SQL.