Skip to content
Docs

Reliability and Fault Tolerance

Circuit Breaker: Per-route and per-provider, Closed/Open/Half-Open states, configurable thresholds. Retry Manager: Exponential backoff with jitter, per-error-type retries, retry-after respect. Health Monitoring: Automatic checks, status tracking, auto-failover, priority-based fallback. Graceful Degradation: When a backend provider is degraded, requests automatically shift to the next-priority provider in the same model family; rate-limited callers receive 429 with Retry-After rather than 5xx, so well-behaved SDKs back off cleanly. Self-Healing: SQLite schema drift is detected and backfilled at boot via the schema-sync module (handles every column added since 2.0.20 — auth wizard, SCIM, LDAP, etc.); stale knowledge-extractor jobs are reaped on startup; circuit breakers re-probe Half-Open on a configurable interval. Zero-Downtime Upgrades: Multi-instance deployments behind HAProxy support rolling upgrades — drain a node, upgrade it, return it to the pool; in-flight requests are not interrupted thanks to SIGTERM → 15s grace shutdown (Section 39).