LLM Gateway & Routing
AgentWatch’s gateway is a drop-in replacement for calling a model provider directly. Applications keep the SDK they already use, and AgentWatch decides which provider actually serves the request — which is what makes provider choice an operational decision rather than a code change.
Which providers and models are supported?
Section titled “Which providers and models are supported?”14 providers with 300+ pre-configured models: OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure OpenAI, Groq, Mistral, Cohere, Together AI, OpenRouter, Fireworks AI, DeepSeek, Ollama, and any custom OpenAI-compatible endpoint. Self-hosted engines — including Lifeboat, vLLM, TGI, Triton, and llama.cpp — connect through the custom or Ollama provider type with only a base-URL change.
The model catalog carries context length, maximum output, capabilities (vision, function calling, JSON mode, streaming, tool use, reasoning), and per-model input/output pricing, with automatic pricing updates and model discovery from provider APIs. Administrators control which models are exposed to which teams, can alias models per organization, and can retire a model without breaking existing API consumers.
What API surface does the gateway expose?
Section titled “What API surface does the gateway expose?”A full OpenAI-compatible surface — chat completions, legacy completions, embeddings, images, audio, batches, reranking, fine-tuning, moderations, and the complete Assistants, Files, and Vector Stores APIs — plus native endpoints for Anthropic Messages, Google Vertex AI, Azure OpenAI, and Cohere V2.
Can an app written for one provider call a different one?
Section titled “Can an app written for one provider call a different one?”Yes — this is AgentWatch’s answer to vendor lock-in. The inbound SDK format is decoupled from the outbound provider: an application written against the OpenAI SDK can be routed to Anthropic, AWS Bedrock, or Google Vertex AI, with the request and response bodies translated in both directions. OpenAI, Vertex AI, Azure OpenAI, and Cohere V2 are supported as inbound formats and can reach any of the 14 upstream providers.
The practical consequence: providers can be swapped during an outage, for cost reasons, or to satisfy a regional sovereignty requirement without changing application code.
How does routing decide where a request goes?
Section titled “How does routing decide where a request goes?”Routing is configurable per route or globally, and the modes compose with policy:
| Mode | Behavior |
|---|---|
| Semantic | Route by request topic or intent, matched on embedding similarity |
| Weighted | Split traffic across model variants by percentage |
| Round-robin | Distribute across keys and providers in the same model family |
| Least-loaded | Pick the key with the lowest in-flight count |
| Cost-optimized | Prefer the cheapest model meeting the required capabilities |
| Priority / QoS | Latency-sensitive traffic jumps ahead of bulk batch work |
| Failover | Health-monitored switch to an alternate when the primary fails or exceeds latency thresholds |
Gateway routes additionally support glob path matching, header-based routing, body and path transformation, content-type filtering, size limits, per-route IP allowlisting, per-route authentication, and per-route circuit breakers.
How are provider credentials managed?
Section titled “How are provider credentials managed?”Through key pooling: multiple keys per provider with per-key concurrency and rate limits, team affinity, and load balancing across them. Keys can be probed on demand against the live provider to confirm they still authenticate and have quota, per-key success and latency metrics are tracked so degraded keys are deprioritized automatically, and an explicit priority order determines failover. Rotation is scheduled or on demand, with alerts before expiry.
What happens when a provider fails?
Section titled “What happens when a provider fails?”Circuit breakers trip per route and per provider; retries use exponential backoff with jitter and respect provider retry-after headers; health checks drive automatic failover to the next-priority provider in the same model family; and rate-limited callers receive a proper 429 with Retry-After rather than a server error, so well-behaved SDKs back off cleanly instead of retrying into the wall.
Does caching reduce cost?
Section titled “Does caching reduce cost?”Yes, three layers:
- Semantic cache — vector-similarity matching serves a cached response for a semantically equivalent question without calling the provider at all, with a configurable similarity threshold and TTL. Typical savings of 20–40% on repetitive workloads like FAQ support and code completion.
- Provider-native prompt caching — passed through transparently and enabled by default with no configuration, including Anthropic cache breakpoints (documented at up to 90% cost reduction on cache hits for large system prompts and tool definitions) and OpenAI automatic prompt caching, with cached-token counts surfaced in usage reports.
- Response cache — exact-match caching for deterministic requests.
Sources and references
Section titled “Sources and references”- Budget and quota enforcement on this traffic: Governance and cost control.
- Connecting private models: Lifeboat overview.