Scalability and Performance

Single-instance peak throughput: ~1.5–2K light req/sec, ~5–10K WebSocket, ~500–1K MCP sessions, ~200–400 files/sec parsing. Multi-instance (PG + Redis + LB): 3x ~2.4K req/sec, 6x ~4.8K, 12x ~9.6K. Egress tunnel: per-request long-poll NDJSON, concurrency lifted in v1.0.75 for chatgpt.com to unblock the agent event loop.

40.1 Throughput reference by user count

These reference points are for I/O-bound LLM workloads — actual throughput is gated by upstream provider latency, not AgentWatch CPU.

Users	Concurrent sessions	Requests / min (est.)	Requests / sec (est.)	Bottleneck
100	5–20	300–600	5–10	Upstream LLM API
500	25–100	1.5K–3K	25–50	Upstream / DB
1,000	50–200	3K–6K	50–100	App concurrency / DB
5,000+	250–1,000	15K–30K	250–500	Scale horizontally — add app replicas behind LB

40.2 Built-in capacity limits (configurable)

Parameter	Default	Source
Max concurrent outbound requests	500	RequestQueue
Max queued requests (backpressure)	10,000	maxQueueSize
Queue timeout	60 seconds	requestTimeout
Outbound socket pool per host	1,000	HTTP agent
Per-API-key concurrency	5 (configurable)	per-key
Per-API-key req/min	60 (configurable)	per-key
Per-tenant req/min	1,000 (configurable)	per-tenant
Token bucket refill rate (global)	100 tokens/sec	TokenBucket
Token bucket max burst	1,000 tokens	TokenBucket
LLM call timeout	5 minutes	axios
Max request body	50 MB	express
Max response	100 MB	axios
Circuit breaker trip threshold	5 failures	CircuitBreaker
Circuit breaker recovery window	60 seconds	recoveryTimeout
Retry attempts on failure	3 (exponential backoff)	RetryManager

40.3 Memory pressure and OOM diagnosis

Bun’s JS heap is small relative to total process RSS — most memory lives in native buffers (LanceDB Rust handles, bun:sqlite, HTTP/2 buffers, Bun’s allocator). When the container is cgroup-OOM-killed at the configured memory: limit, the gap between heapUsed and rss tells you whether the leak is in JS or native code:

1. GET /admin/diagnostics/memory — returns process.memoryUsage() + uptime. If rss is 10–20× heapUsed, the leak is native.
2. POST /admin/diagnostics/heap-snapshot — writes a V8 snapshot under $DATA_DIR for docker cp retrieval. Only useful when JS heap is the dominant contributor.
3. Inside the container: dmesg | grep -i "killed process" confirms cgroup OOM and prints the anon-rss at kill time.

Known native-memory hardening (v2.0.63):

• LanceDB Table handles are pooled per-connection in SessionVectorStore — previously a fresh handle was allocated per embedSession call, pinning Rust buffer pools the JS GC couldn’t see.
• The Python bridge (Presidio NER) matches responses by _requestId rather than always resolving the first pending entry — the old behaviour misrouted responses under concurrency and left correctly-keyed pending entries retaining their full request JSON in closures until 30 s timeout.
• Rate-limit maps (bandwidthTracker, agentProxyRateLimits, mcpProxyBuckets) have periodic idle-entry eviction so they cannot grow unbounded with one entry per ever-seen userEmail.

40.4 High availability

• App server — stateless, JWT sessions, no sticky load balancing required
• Horizontal scale-out — add app replicas behind HAProxy / ALB / NLB; linear scale
• Database HA — Postgres primary + read replicas; PgBouncer connection pooling beyond 1K users
• Provider failover — automatic via circuit breaker (5 failures → open, recover after 60s)
• Region DR — replicate the entire pod, route via DNS or global LB; RPO < 5 min via Postgres WAL streaming + S3 archive; RTO < 30 min

---