Skip to content
Docs

What's New in 2.0.21–2.0.105

This section consolidates the major changes since the previous 2.0.20 reference snapshot. §42.1–42.10 cover 2.0.21–2.0.63; §42.11 covers 2.0.64–2.0.105.

42.1 Identity (largest change set)

Section titled “42.1 Identity (largest change set)”
  • Native SAML 2.0 (@node-saml/node-saml) — SP metadata, AuthnRequest, ACS callback, SLO; signed assertions with SHA-256
  • LDAP / Active Directory authentication (ldapts) with LDAPS, CA cert upload, skip-verify TLS option, super-admin local fallback, host normalization, edit endpoint
  • LDAP target-org routing — provider default + domain → org rules (LdapOrgRoutingRule)
  • Group-to-team mapping unified across OAuth2/OIDC, SAML, and LDAP via the SsoGroupMapping table and applyGroupMappings() pipeline. Modes: Off / Auto / Mapped, with Auto-mode rows synthesized on the fly
  • Unified Auth Wizard — hub-and-spoke admin modal with brand-logo vendor tiles and declarative protocol descriptors; exposes Auto-create + Enabled toggles on OIDC/SAML
  • SCIM v2 rewrite — per-org mount (/scim/v2/:orgId/…), per-org ScimConfig, hashed ScimToken with reveal-once generation modal, group sync mode (Off/Auto/Mapped), discovery endpoints, full Users + Groups CRUD with PATCH dispatch
  • User API keys unified to ak_… prefix as the sole user-key format; Company Settings format respected on provision and rotation
  • Auto-created teams now seed budgets from the org’s default budget template

42.2 Endpoint Agent (AgentWatch Connect)

Section titled “42.2 Endpoint Agent (AgentWatch Connect)”
  • Curl-impersonate-chrome bundled and signed in macOS, Linux, and Windows installers (vendored v1.5.6)
  • Egress tunnel — long-poll NDJSON channel routes chatgpt.com / claude.ai / gemini fetches through the user’s machine; Chrome TLS+H2 fingerprint matched
  • CRL Distribution Point served by the local agent’s setup-server so CDP URLs work cross-platform
  • Heartbeat telemetry — curl-impersonate variant usage + coverage-gap suppress list
  • Linux hardening + ChatGPT anonymous logging
  • Windows reliability — NSSM-aware daemon detection, proxy detection on 127.0.0.1:443, ACL re-application after NSSM log creation, ~30x faster isDaemonRunning probe, BUN_OPTIONS on the NSSM service, faster install elevation
  • DNS fallback to 1.1.1.1 with visible error; matches the proxy’s c-ares workaround

42.3 Platform / Runtime

Section titled “42.3 Platform / Runtime”
  • Bun pin bumped to 1.3.14 (HTTP/2 client + image API + faster installs)
  • Default local TLS listener: Bun.serve — HTTP/1.1 + HTTP/3 (QUIC over UDP/443) on the same port; https.createServer is now opt-out via AGENTWATCH_PROXY_BUNSERVE=0 (commits 0e6b678b opt-in → 01f9bb7e default-on)
  • HTTP/3 hardening on every response path — Alt-Svc + Alternate-Protocol stripped from upstream responses so browsers don’t cache the QUIC endpoint and bypass the TCP-only MITM (proxy.ts, enhanced-index.ts, api-routes.ts)
  • bcryptjs replaces native bcrypt (Bun 1.3 + Node 24 ABI 137 compatibility)
  • Comprehensive SQLite self-healing migrations at boot (auth wizard, SCIM, retired tables)
  • maxApiCallsPerMonth=0 now correctly means unlimited

42.4 Schema (44 models total)

Section titled “42.4 Schema (44 models total)”

New tables since 2.0.20:

  • LdapProvider, LdapOrgRoutingRule, SsoGroupMapping
  • ScimConfig, ScimToken (+ users.externalId)
  • New columns on OAuth2Provider: protocol, samlEntryPoint, samlIssuer, samlCertificate, samlCallbackUrl, groupClaimKey, brand, logoUrl, draftAt, groupMappingMode, autoTeamParseRule, autoDefaultRole, autoDefaultBudgetTemplateId

42.5 Fixes worth calling out

Section titled “42.5 Fixes worth calling out”
  • DLP fullName no longer false-positives on factual queries like “what is the name of X”
  • Chat history view renders clean text (no raw JSON, no truncation surprises)
  • Super-admin team creation defaults to Platform org when no org specified
  • Super-admin orgId fallback resolved on SCIM and SSO admin routes
  • PowerShell scripts standardized to ASCII (no em-dashes / checkmarks — PS 5.1 Windows-1252 default broke them twice)

42.6 Storage architecture overhaul (v2.0.45+)

Section titled “42.6 Storage architecture overhaul (v2.0.45+)”

The single biggest reliability change in the 2.0.45–2.0.63 window. Production deployments grew the usage SQLite to 12+ GB in days because two write paths (transparent-proxy, agent-web-proxy) inlined full request + response payloads into usage.metadata — averaging ~2 MB per row, max 13 MB. That bloat made /admin/analytics/tasks aggregate slowly, pinned big SQLite working buffers, and contributed to recurring cgroup OOMs.

  • Request body archive (new — see §35.1). Full request/response bodies now write to disk as AES-256-GCM-encrypted, gzip-compressed files at {File-Archiver-base-path}/{org}/bodies/YYYY/MM/DD/HH/{requestId}.json.gz.enc — same volume as the File Archiver Default Sink, so customer NFS / EBS / S3FS mounts are honored transparently. DB row carries only a bodyArchive: { path, base_path, recorded_at, encrypted: true } pointer (~1–2 KB total metadata).
  • Backfill migration runs from start.sh on every docker compose up -d — idempotent walk of usage rows with LENGTH(metadata) > 50000. Bodies are extracted into the encrypted archive using each row’s original timestamp (so directory tree and file mtime match request time), inline fields are stripped, and VACUUM reclaims disk. Production DBs shrink from 12 GB to a few hundred MB.
  • Daily retention sweep (new — see §35.1, src/enhanced/storage/body-archive-retention.ts). Deletes encrypted body files older than each org’s eventPipelineFileArchiverRetentionDays (default 90). Empty directories are unwound on the way back up. First sweep ~5 min after boot to avoid competing with startup migrations.
  • Vector store keeps full-content embeddings. SessionVectorStore.embedSession() gained promptOverride / responseOverride parameters so the two refactored call sites can feed full in-memory content directly. Semantic search quality does not regress despite bodies no longer being in DB.
  • Encryption key sourced from BODY_ARCHIVE_KEY_HEX env var, falling back to CHAT_ENCRYPTION_KEY, then to the same hardcoded material ChatFileStore uses. SHA-256 to 32-byte AES-256 key. File format: [ IV(12) ][ authTag(16) ][ ciphertext = AES-256-GCM( gzip( JSON ) ) ].

42.7 Model catalog routing — FK over string (v2.0.45+)

Section titled “42.7 Model catalog routing — FK over string (v2.0.45+)”

models.providerId (FK → providers.id) is now the source of truth for routing. The legacy models.provider string column is kept for display only. At request time selectProviderForModel resolves through the FK, falling back to the string only for legacy rows where providerId is null. This prevents UI renames and per-org provider-name drift from silently breaking /v1/chat/completions while /v1/models keeps listing the model.

  • Startup reconciliation in start.sh: FK-set rows whose provider string disagrees with the FK target are logged (no data mutation — runtime routes via FK). Orphan rows (FK null + string unmatched in the row’s org) auto-repair when the org has exactly one kind='openai-compatible' provider enabled.
  • Cross-org safety: the FK lookup verifies the resolved Provider record belongs to the same org as the calling tenant before accepting it — prevents data leaks via cloned catalog rows that point at another org’s Provider id.

42.8 Native-memory leak hardening (v2.0.45+)

Section titled “42.8 Native-memory leak hardening (v2.0.45+)”

Eliminated three sources of unbounded native memory growth in the bun process. Reduced observed OOM cadence from every ~27 min under load to steady-state.

  • LanceDB Table handles pooled in SessionVectorStore — previously a fresh native handle was allocated per embedSession call, pinning Rust buffer pools the JS GC could not see.
  • Python bridge id-based response matching — PythonBridge.handleResponse (Presidio NER IPC) now matches responses by their echoed _requestId instead of always resolving the first pending entry. The old behavior misrouted responses under concurrency and left correctly-keyed pending entries retaining their full request JSON in closures until the 30 s timeout.
  • Rate-limit map eviction — bandwidthTracker, agentProxyRateLimits, mcpProxyBuckets now have periodic idle-entry sweeps (6 h / 30 m / 30 m) so they can’t accumulate one entry per ever-seen userEmail indefinitely.

42.9 Diagnostics endpoints (v2.0.45+)

Section titled “42.9 Diagnostics endpoints (v2.0.45+)”

Two superadmin endpoints for live triage of memory issues without needing to attach a debugger:

  • GET /admin/diagnostics/memory — process.memoryUsage() + uptime. Cheap, pollable. When rss >> heapUsed (e.g., 2.8 GB vs 170 MB), the leak is native, not JS-side.
  • POST /admin/diagnostics/heap-snapshot — writes a V8 heap snapshot under $DATA_DIR/heap-<ts>.heapsnapshot for docker cp retrieval and Chrome DevTools “Compare” diffing. Only useful when JS heap is the dominant contributor.

See §40.3 for the OOM diagnosis playbook.

42.10 Endpoint Agent updates (Connect v1.0.78 – v1.0.93)

Section titled “42.10 Endpoint Agent updates (Connect v1.0.78 – v1.0.93)”
  • Multiple agents sharing one API key no longer collapse into a single Agent Fleet record (v1.0.92 server fix)
  • Linux fail-open on EACCES /etc/hosts fixed — previously cascaded into overload + dead UI when the agent could not write to /etc/hosts
  • Firefox trusts the AgentWatch CA on macOS + Windows via ImportEnterpriseRoots enterprise policy (v1.0.90)
  • macOS Sonoma+ dropped its SIGHUP mDNSResponder DNS-flush — agent now uses plain SIGTERM (launchd respawns) so /etc/hosts changes actually take effect
  • Signed installers — Apple Developer ID + notarization on macOS, Authenticode on Windows (v1.0.93)

42.11 What’s New in 2.0.64–2.0.105

Section titled “42.11 What’s New in 2.0.64–2.0.105”

The 2.0.64–2.0.105 window focused on multi-tenant URL routing, sub-path deployment, the A2A protocol, redaction accuracy, and user-provisioning correctness.

A2A (Agent-to-Agent) protocol — now implemented (was design-spec-only at 2.0.63)

  • Agent registry with admin CRUD (/api/admin/a2a/agents, create / update / rotate-key / disable), backed by the new A2AAgent and A2AMessage tables
  • A2A-spec-compliant agent card discovery — GET /a2a/.well-known/agent.json/:agentId
  • JSON-RPC 2.0 endpoint — POST /a2a/v1/:agentId (tasks/send, tasks/get, tasks/cancel, tasks/sendSubscribe)
  • Agent-side message polling + reply (/a2a/v1/queue/:agentId, /a2a/v1/messages/:messageId/reply) and intra-tenant discovery (/a2a/v1/agents/discover)
  • Authentication via X-A2A-Key header + Bearer JWT; full organization scoping / tenant isolation
  • Deferred to a later phase: mTLS workload identity, RFC 8693 token exchange, push-notification webhooks, external-agent egress proxy (see §43.1)

Multi-tenant URL routing — bare-slug scheme

  • Auth APIs migrated to root /api/auth/*; per-tenant access via a bare /<slug>/… path scheme with role-disambiguated /platform/* routes mounted unconditionally
  • Expanded reserved-slug blacklist; runtime skips reserved slugs at lookup so probe/health URLs don’t collide with tenant slugs
  • Legacy redirects for the old /o/<slug>/ scheme (301 on login, 410 on retired auth paths, /admin/* → /<jwt.slug>/admin/*)
  • Editable workspace slug in Company Settings (validated); super_admin can change a tenant’s slug and reassign the tenant admin
  • safeReturnPath enforces the /<slug>/ prefix; SSO post-callback redirect routes through sso-callback.html to persist JWT to localStorage; lazy self-heal of user_email_index on miss

Base path / sub-path deployment

  • Setting BASE_PATH (e.g. /agentwatch) now routes all LLM chat, gateway, MCP, SSE, WebSocket, and UI traffic through the sub-path; the prefix is stripped internally before route matching (no nginx rewrite-target needed)
  • Global window.fetch override + window.getAbsoluteUrl injected into every HTML response so static JS honors the base path with no per-file edits; raw server.on('upgrade') WebSocket handler strips the prefix; knowledge UI index.html routed through the base-injection function; Vite base parameterized via VITE_BASE
  • Helm chart wired — BASE_PATH exposed as a container env var (values.yaml), liveness/readiness/startup probes auto-prefix the health path via the agentwatch.healthPath helper, ingress path guidance documented inline

DLP / redaction accuracy (resolves reported 403 false-positives)

  • Classifier sensitivity and default policy thresholds tuned — benign prompts (arithmetic, general questions, city names) no longer mis-flagged as restricted / personal information
  • Tool-level redaction fixed — ADK / web_search tool output is no longer over-blocked
  • Blocked requests now return a user-friendly block response instead of NETWORK_ERROR; PII scrubbed from stored preview + chat history on blocked requests; riskScore + classifications recorded on redacted (not just blocked) requests
  • System LLM analysis streamed; generic OpenAI-compatible routing for compliance analysis

Model management

  • “Add Model” pop-up and Vertex AI model listing fixed
  • Platform-org consolidation — platform models stamped with the platform org id (not NULL); removed the hardcoded shared platform-org seed id

User provisioning

  • Every user creation path (SSO approval, SCIM, admin add) now ensures a Tenant + api_key
  • Orphan-tenant reuse on email collision (no more P2002 unique-constraint errors)
  • super_admin selects the target Organization when adding an employee; slug + email-domain uniqueness enforced on tenant create
  • LDAP {{username}} / {{username_local}} / {{email}} template tokens; owning email-domain derived from searchBase / bindDN; route discovery by LDAP provider domain restriction; super_admin break-glass bypass of LDAP enforcement at tenant login

Security / dependencies

  • Protobuf updated to 7.5.6 (NPM vulnerability resolved)
  • SSO hardening — token-in-URL replaced with a single-use ticket + client-side return validation; nuanced email_verified policy for Entra compatibility
  • Egress: ALPN h1 fallback + heartbeat passthrough for body-less ChatGPT endpoints

Rate limiting / usage / build

  • Rate limiting reads the platform-wide row only (dropped the per-org Math.min clamp)
  • Routing config lazily reloads per-org on a real cache miss
  • organization_id written on every usage row + historical NULL backfill
  • Build: bun --compile server binary option (--server-binary), trimmed Docker build context, scripts/ copied into the production image for boot migrations, env-overridable super admin credentials (SUPER_ADMIN_EMAIL)