LLM Gateway & Routing

AgentWatch’s gateway is a drop-in replacement for calling a model provider directly. Applications keep the SDK they already use, and AgentWatch decides which provider actually serves the request — which is what makes provider choice an operational decision rather than a code change.

Which providers and models are supported?

15 providers with 300+ pre-configured models: OpenAI, Anthropic, Google Vertex AI, AWS Bedrock, Azure OpenAI, Azure AI Foundry, Groq, Mistral, Cohere, Together AI, OpenRouter, Fireworks AI, DeepSeek, Ollama, and any custom OpenAI-compatible endpoint. Self-hosted engines — including Lifeboat, vLLM, TGI, Triton, and llama.cpp — connect through the custom or Ollama provider type with only a base-URL change.

The model catalog carries context length, maximum output, capabilities (vision, function calling, JSON mode, streaming, tool use, reasoning), and per-model input/output pricing, with automatic pricing updates and model discovery from provider APIs. Administrators control which models are exposed to which teams, can alias models per organization, and can retire a model without breaking existing API consumers.

What API surface does the gateway expose?

A full OpenAI-compatible surface — chat completions, legacy completions, embeddings, images, audio, batches, reranking, fine-tuning, moderations, and the complete Assistants, Files, and Vector Stores APIs — plus native endpoints for Anthropic Messages, Google Vertex AI, Azure OpenAI, and Cohere V2.

Can an app written for one provider call a different one?

Yes — this is AgentWatch’s answer to vendor lock-in. The inbound SDK format is decoupled from the outbound provider: an application written against the OpenAI SDK can be routed to Anthropic, AWS Bedrock, or Google Vertex AI, with the request and response bodies translated in both directions. OpenAI, Vertex AI, Azure OpenAI, and Cohere V2 are supported as inbound formats and can reach any of the 15 upstream providers.

The practical consequence: providers can be swapped during an outage, for cost reasons, or to satisfy a regional sovereignty requirement without changing application code.

How does routing decide where a request goes?

Routing is configurable per route or globally, and the modes compose with policy:

Mode	Behavior
Semantic	Route by request topic or intent, matched on embedding similarity
Weighted	Split traffic across model variants by percentage
Round-robin	Distribute across keys and providers in the same model family
Least-loaded	Pick the key with the lowest in-flight count
Cost-optimized	Prefer the cheapest model meeting the required capabilities
Priority / QoS	Latency-sensitive traffic jumps ahead of bulk batch work
Failover	Health-monitored switch to an alternate when the primary fails or exceeds latency thresholds

Gateway routes additionally support glob path matching, header-based routing, body and path transformation, content-type filtering, size limits, per-route IP allowlisting, per-route authentication, and per-route circuit breakers.

How does complexity routing pick a model, and what is the RAG tier?

Send "model": "auto" and AgentWatch judges how hard the prompt is, then routes it to the model you mapped to that tier. It is off by default; turn it on at Settings → Complexity Routing.

Three things judge the prompt, in order, and the first confident answer wins:

Tier	What it is	Cost
Heuristics	length, code markers, tool use, context size	free
Classifier	a trained ONNX model predicting task, domain, complexity 1–5 and language	~70–90ms, flat with prompt size
RAG tier	embeds your prompt and votes from the nearest labelled seed examples	single-digit ms

The RAG tier is a second opinion, used when the classifier is unsure or absent. The embedding model (all-MiniLM-L6-v2, int8, ~22 MB) ships inside AgentWatch and runs locally in the inference worker — no external embedding service and nothing leaves the gateway. Seed vectors are precomputed at build time, so each request embeds only the prompt. Its advantage over the classifier is that you adapt it by adding seed examples rather than retraining.

⚠ The classifier reads at most the first 256 tokens of a prompt, so its cost is effectively constant — a 20-word prompt and a 60,000-word one measure within 20ms of each other. Routing never becomes expensive on long inputs. Every release verifies this against a 3 second per prompt ceiling on a live deployment.

What if a tier lists more than one model?

Write them comma-separated — modelA,modelB for an equal split, or modelA:3,modelB:1 to weight them. Multi-upstream strategy then decides how one is picked:

Strategy	Behaviour
Failover (primary → chain)	always sends to the first model; if it is unreachable the request is retried down the rest of the list, in order
Round-robin	rotates every request
Weighted	splits by the ratio you wrote
Random (A/B)	picks one per request

A tier with a single model ignores this setting entirely.

⚠ What failover retries, and what it does not. Another model is tried for a 5xx, a timeout, a connection failure, a 429 or a 404 — the model or endpoint is unwell, or saturated, or not served here. It is not retried for an ordinary 4xx: a 400 means the request itself is wrong, and the next model would be handed the identical request and refuse it identically, costing latency and quota to reach the same answer. Retries happen before any bytes reach your client, so you never see a half-written answer, and are capped at 3 attempts per request (AW_ROUTER_FAILOVER_MAX_ATTEMPTS) so a tier listing a dozen models cannot turn one call into a dozen.

The route taken is reported on every response:

x-agentwatch-route: coding/science/c3/classifier/map[3]/failover->modelA
x-agentwatch-route-chain: modelA,modelB

⚠ Changes are not live until you press Save routing config. Flipping a toggle marks the panel as having unsaved changes and says so above the button.

How are provider credentials managed?

Through key pooling: multiple keys per provider with per-key concurrency and rate limits, team affinity, and load balancing across them. Keys can be probed on demand against the live provider to confirm they still authenticate and have quota, per-key success and latency metrics are tracked so degraded keys are deprioritized automatically, and an explicit priority order determines failover. Rotation is scheduled or on demand, with alerts before expiry.

What happens when a provider fails?

Circuit breakers trip per route and per provider; retries use exponential backoff with jitter and respect provider retry-after headers; health checks drive automatic failover to the next-priority provider in the same model family; and rate-limited callers receive a proper 429 with Retry-After rather than a server error, so well-behaved SDKs back off cleanly instead of retrying into the wall.

Does caching reduce cost?

Yes, three layers:

• Semantic cache — vector-similarity matching serves a cached response for a semantically equivalent question without calling the provider at all, with a configurable similarity threshold and TTL. Typical savings of 20–40% on repetitive workloads like FAQ support and code completion.
• Provider-native prompt caching — passed through transparently and enabled by default with no configuration, including Anthropic cache breakpoints (documented at up to 90% cost reduction on cache hits for large system prompts and tool definitions) and OpenAI automatic prompt caching, with cached-token counts surfaced in usage reports.
• Response cache — exact-match caching for deterministic requests.

How do I point an application at AgentWatch?

Change the base URL and the key. Nothing else:

export OPENAI_BASE_URL="https://<your-agentwatch-host>/v1"
export OPENAI_API_KEY="ak_…"

from openai import OpenAI
client = OpenAI(base_url="https://<your-agentwatch-host>/v1", api_key="ak_…")
client.chat.completions.create(model="gpt-4o", messages=[{"role":"user","content":"hi"}])

Anthropic clients use /v1/messages on the same host with the same key. A request that works directly against the provider works here unchanged — the proxy forwards the whole body, including parameters it does not itself interpret.

How do I add a provider and its models?

1. Settings → Providers → Add provider, choose the type and paste the key.
2. Fetch models — the catalogue is populated from the provider’s own list, with context length, capabilities and pricing.
3. Expose the ones you want to each team; a model absent from a team’s list is not callable by it.

curl -s -X POST https://<your-agentwatch-host>/admin/providers/fetch-models \
  -H "Authorization: Bearer <tenant-admin JWT>" -H 'Content-Type: application/json' \
  -d '{"providerId":"<provider id>"}'

curl -s https://<your-agentwatch-host>/admin/models/catalog \
  -H "Authorization: Bearer <tenant-admin JWT>"

Refreshing re-syncs from the provider, which will overwrite a context length an administrator typed by hand — an admin-set window is pinned and respected by the sync, but everything else comes from the provider.

How do I connect Azure AI Foundry?

Azure AI Foundry is its own provider type, separate from Azure OpenAI. One Foundry resource serves many model families — Azure OpenAI, DeepSeek, Grok, Llama, Mistral, Phi and others — and each is reached through a deployment. The deployment name is the model ID: it is what callers send as model, exactly as you typed it in the Foundry portal (it is case-sensitive).

Validation status. This provider type is built against Microsoft’s published Foundry v1 API specification. It has not yet been validated against a live Foundry resource. Run the verification step below before routing production traffic to it, and tell us if anything differs.

AgentWatch calls Foundry’s OpenAI-v1-compatible surface:

Request URL	https://<resource>.services.ai.azure.com/openai/v1/chat/completions (also /embeddings, /responses, …)
Authentication	api-key: <resource key> header
api-version	not sent — v1 defaults to the current GA surface
model	your deployment name, sent verbatim

The older Azure AI Model Inference endpoint (…/models/chat/completions?api-version=2024-05-01-preview) is deprecated by Microsoft and is not used. If you paste it, or the .openai.azure.com host of the same resource, or a Foundry project endpoint (…/api/projects/<name>), AgentWatch normalises it to the v1 base.

Console: API Key Management → Add Provider → Azure AI Foundry.

1. Base URL — the resource endpoint from Foundry portal → your resource → Keys and Endpoint, e.g. https://contoso-ai.services.ai.azure.com.
2. API Key — KEY 1 or KEY 2 from the same page. Test Key checks it against GET /openai/v1/models.
3. Deployment names — comma-separated, from Models + endpoints, e.g. gpt-4.1-mini, DeepSeek-R1. These are not auto-detected: Foundry’s model listing returns the models the resource can deploy, not the deployments you made, so a fetched list would register names no request can reach.
4. Add Provider, then expose the deployments to teams as with any model.

Adding a provider requires a company admin on a plan that includes API Key Management; any other caller gets 403.

The same thing through the API:

# 1. Create the provider and register its deployments
curl -s -X POST https://<your-agentwatch-host>/api/providers \
  -H "Authorization: Bearer <company-admin JWT>" -H 'Content-Type: application/json' \
  -d '{"name":"Azure AI Foundry","kind":"azure-ai-foundry",
       "baseURL":"https://contoso-ai.services.ai.azure.com",
       "modelIds":["gpt-4.1-mini","DeepSeek-R1"]}'

# 2. Attach the resource key (use the id returned by step 1)
curl -s -X POST https://<your-agentwatch-host>/api/providers/<provider id>/keys \
  -H "Authorization: Bearer <company-admin JWT>" -H 'Content-Type: application/json' \
  -d '{"key":"<Foundry KEY 1>","label":"foundry-key-1"}'

Verify with a gateway key, using a deployment name as the model:

curl -s https://<your-agentwatch-host>/v1/chat/completions \
  -H "Authorization: Bearer ak_…" -H 'Content-Type: application/json' \
  -d '{"model":"DeepSeek-R1","messages":[{"role":"user","content":"Reply with OK"}]}'

You should get a normal chat completion, and the call should appear in Usage Insights under the Azure AI Foundry provider. A gateway route (super admin) of type Azure AI Foundry forwards the caller’s Authorization header only, so inject the key with a route header {"api-key":"<key>"}.

Symptom	Cause	Fix
Test Key: Invalid API key or insufficient permissions	Wrong key, or a key from a different resource	Copy KEY 1 from the same resource’s Keys and Endpoint page
Test Key: No Azure AI Foundry v1 API at … (404)	The base URL is not a Foundry / Azure AI Services resource endpoint	Use https://<resource>.services.ai.azure.com
403 MODEL_NOT_AVAILABLE_FOR_TENANT	The deployment is not exposed to the caller’s team	Check /api/models/available with that key
Upstream 404 DeploymentNotFound	The model is not a deployment name on that resource (or differs in case)	Use the name shown under Models + endpoints, exactly
403 when adding the provider	Not a company admin, or the plan excludes API Key Management	Ask a company admin; check Plans and feature access
A deployment you added is missing after a refresh	Not expected — Foundry rows are never removed by the provider sync	Report it; re-add the name under the provider

How do I check which model a caller may use?

curl -s https://<your-agentwatch-host>/api/models/available \
  -H "Authorization: Bearer ak_…"

That is the list from the key’s own point of view, which is the fastest way to settle “why does this model 403 for them and not for me”.

How do I measure a request before sending it?

curl -s -X POST https://<your-agentwatch-host>/v1/messages/count_tokens \
  -H "Authorization: Bearer ak_…" -H 'Content-Type: application/json' \
  -d '{"model":"<your model>","messages":[{"role":"user","content":"…"}]}'

It returns the input token count as the serving engine sees it — including the injected system prompt and templated tool schemas, which are invisible in your payload and are what makes local estimates wrong — plus the context window and how much output room is left.

Common gateway problems

Symptom	Cause	Fix
403 model not available	The model is not exposed to that team, or the alias does not resolve	Check /api/models/available with that key
400 CONTEXT_LENGTH_EXCEEDED	Input exceeds the window	Shorten it, or use a larger-window model. This is terminal — retrying cannot help
Repeated identical retries	An SDK treating a 5xx as retryable	Overflow and policy refusals are now 4xx; if you see 5xx, it is a genuine upstream fault
Latency spikes on one provider	Key pool contention or a degraded key	Per-key latency and success metrics are on the provider page; deprioritize or rotate
Responses look like a different model	A pooled upstream served it	The upstream_endpoint tag on the usage row records which engine answered

Sources and references

• Budget and quota enforcement on this traffic: Governance and cost control.
• Connecting private models: Lifeboat overview.
• Azure AI Foundry v1 API (Microsoft Learn): https://learn.microsoft.com/en-us/azure/foundry/openai/api-version-lifecycle
• Migrating from the Model Inference API to v1 (Microsoft Learn): https://learn.microsoft.com/en-us/azure/foundry/how-to/model-inference-to-openai-migration