Model Registry and Provider Management
4.1 Supported LLM Providers (14)
Section titled “4.1 Supported LLM Providers (14)”| Provider | Protocol | Notes |
|---|---|---|
| OpenAI | OpenAI-compatible | 50+ models pre-configured |
| Anthropic | Native SDK | Claude family (Claude 4.5/4.6/4.7, Sonnet, Opus, Haiku) |
| Google Vertex AI | Google SDK | Gemini 1.5/2.0/2.5/3 family |
| AWS Bedrock | AWS SDK | Multi-model |
| Azure OpenAI | OpenAI-compatible | Deployment-based |
| Groq | OpenAI-compatible | Pre-configured |
| Mistral | OpenAI-compatible | Pre-configured |
| Cohere | Cohere V2 + OpenAI-compatible | Native V2 endpoint |
| Together AI | OpenAI-compatible | Pre-configured |
| OpenRouter | OpenAI-compatible | 100+ models |
| Fireworks AI | OpenAI-compatible | Pre-configured |
| DeepSeek | OpenAI-compatible | Pre-configured |
| Ollama | REST API | Self-hosted |
| Custom | OpenAI-compatible | Any OpenAI-spec endpoint |
Self-hosted servers (vLLM, TGI, Triton, llama.cpp, Ollama) work via the Custom or Ollama provider type — no application code changes needed, only a base-URL swap. Open-source families covered: LLaMA, Mistral, Gemma, Phi, Qwen, DeepSeek, Yi, Falcon (whatever the underlying inference server exposes on its OpenAI-compatible endpoint).
4.2 Model Catalog
Section titled “4.2 Model Catalog”- 300+ pre-configured models with metadata (context length, max output tokens, capabilities, pricing)
- Per-model pricing: input/output costs per 1K and per 1M tokens
- Capability tracking: vision, function_calling, json_mode, streaming, tool_use, reasoning
- Dynamic model discovery from provider APIs
- LiteLLM pricing integration (automatic updates)
- Organization-scoped models — administrators provision which models are exposed to which workspaces/teams via
ModelAccess - Per-model aliases per organization (workspace-level renaming for governance)
- Provider templates for one-click setup
- Model lifecycle: enable/disable to retire models without breaking existing API consumers
4.3 API Key Pooling
Section titled “4.3 API Key Pooling”Multiple keys per provider: concurrency (default: 5), maxRequestsPerMinute (default: 60), enable/disable, team affinity, request/error counters, average response time. Load balancing: round-robin, least-loaded, weighted.
Per-key efficacy testing: each provider key can be probed on demand against the live provider to verify it still authenticates and has quota remaining. Per-key performance metrics (success rate, mean latency, error rate by error class) are tracked continuously so degraded keys can be auto-deprioritized.
Failover ranking: keys (and providers) carry an explicit priority ordering — when the primary fails health checks or trips its circuit breaker, the next-priority key in the same model family takes over without dropping in-flight requests (Section 8).
Key rotation: scheduled (90-day default, configurable per key) or on-demand. Rotation alerts fire via email/webhook before expiry.