LLM Gateway and Proxy

3.1 OpenAI-Compatible API

Drop-in replacement for the OpenAI API. Supported endpoints:

• POST /v1/chat/completions — Chat (streaming and non-streaming)
• POST /v1/completions — Legacy completions
• POST /v1/embeddings — Text embeddings
• POST /v1/images/generations, /edits, /variations — Image generation
• POST /v1/audio/transcriptions, /translations, /speech — Audio
• POST /v1/batches — Batch processing (create, list, get, cancel)
• POST /v1/rerank — Document reranking
• POST /v1/fine_tuning/jobs — Fine-tuning (jobs, events, checkpoints)
• POST /v1/moderations — Content moderation
• Full Assistants API (/v1/assistants, threads, messages, runs, steps, tool outputs)
• Full Files API (/v1/files — upload, list, download, delete)
• Full Vector Stores API (/v1/vector_stores — CRUD, files, file batches)
• GET /v1/models — Model listing

Additional Protocol Support:

• POST /v1/messages — Anthropic Messages API
• POST /v1beta/models/:model:generateContent — Google Vertex AI
• POST /v1beta/models/:model:streamGenerateContent — Google Vertex AI streaming
• POST /openai/deployments/:deployment/chat/completions — Azure OpenAI
• POST /openai/deployments/:deployment/embeddings — Azure OpenAI embeddings
• POST /v2/chat — Cohere V2

3.2 Request Processing Pipeline

Every LLM request passes through: (1) Authentication (API key/JWT/SSO), (2) Tenant Mode Resolution (proxied/passthrough/enriched/post_process), (3) Rate Limiting (token-bucket per-user/org/global), (4) DLP Scanning (Presidio NER + custom patterns + webhook DLP), (5) Guardrails (injection/toxicity/content safety), (6) Schema Validation (JSON Schema per route), (7) Model Routing (provider selection, key assignment, circuit breaker), (8) Proxy (streaming, TTFT measurement), (9) Response Processing (tokens, cost, enrichment), (10) Event Emission (Kafka/ClickHouse/file, usage, audit).

3.3 Streaming Support

Full SSE streaming, transparent stream proxying, token counting on streams, Time-to-First-Token (TTFT) measurement, streaming test endpoint for provider validation.

3.4 Integration Methods (Three Paths)

Customers can adopt AgentWatch via any of three integration paths, ordered by friction:

1. Drop-in URL/key swap (lowest friction) — change a single environment variable so the existing OpenAI/Anthropic SDK points at the AgentWatch gateway URL with an AgentWatch-issued API key. Zero code changes. Existing application portfolios onboard in minutes.
2. Native Anthropic API preservation — applications already calling the Anthropic Messages API keep their wire format unchanged; AgentWatch terminates the Anthropic protocol natively and re-emits to whichever upstream provider the routing layer selects.
3. Advanced gateway routes with request/response transformation — for applications that need header injection, path rewriting, body transformation, glob-pattern routing, content-type filtering, or per-route auth, the full Gateway Route configuration (Section 5.1) is exposed.

All three modes preserve full DLP scanning, audit logging, and budget enforcement (Section 6). Mode 1 (drop-in) is the recommended path for fleet-wide rollout because it requires no per-application engineering work.

3.5 Transparent Proxy Mode (Zero-Code Adoption at Scale)

For organizations with large existing AI application portfolios, AgentWatch can operate as a transparent proxy: domain mappings (e.g., api.openai.com → OpenAI provider, api.anthropic.com → Anthropic provider) plus CIDR-based trusted-network validation mean existing apps route through AgentWatch with no code changes whatsoever — not even the URL changes. Credentials can either pass through (audit-only mode) or be replaced at the gateway with org-managed pool keys. Full DLP, audit, budget enforcement, and per-employee attribution all operate in transparent mode. See also Section 5.3 (Zscaler integration) and Section 29 (on-device transparent proxy via AgentWatch Connect).

3.6 Cross-Protocol Translation Matrix (SDK Format Independence)

AgentWatch decouples the inbound SDK format the application uses from the outbound provider the request actually hits. An app written against the OpenAI SDK can talk to AgentWatch and end up routed to Anthropic Claude, AWS Bedrock, or Google Vertex AI — the gateway translates the request body and response body in both directions. This is the foundation of AgentWatch’s vendor-lock-in story (§47.1).

Inbound endpoints (SDK formats AgentWatch accepts):

Endpoint	SDK format	Implementation
POST /v1/chat/completions	OpenAI Chat Completions	Native handler with full DLP / budget / audit pipeline
POST /v1/completions	OpenAI legacy completions	Native handler
POST /v1/messages	Anthropic Messages	Native handler (currently Anthropic-native passthrough)
POST /v1beta/models/:model:generateContent	Google Vertex AI	Native handler via convertOpenAIToVertexNative ↔ convertVertexNativeToOpenAI
POST /v1beta/models/:model:streamGenerateContent	Google Vertex AI streaming	Native handler via convertVertexStreamChunkToOpenAI
POST /openai/deployments/:deployment/chat/completions	Azure OpenAI	Native handler (OpenAI-compatible wire format)
POST /openai/deployments/:deployment/embeddings	Azure OpenAI embeddings	Native handler
POST /v2/chat	Cohere V2 (preamble + chat_history + message)	Native handler — converts Cohere → OpenAI internally, runs the full pipeline (DLP / budget / routing / audit), then converts the response back to Cohere V2 wire format

Outbound provider translation (what the gateway sends upstream):

Upstream provider kind	Translation function	Behaviour
openai	None (pass-through)	Wire format preserved
anthropic	OpenAI → Anthropic Messages (llm-proxy.ts:1577)	System message extracted to top-level system field; image_url content parts converted to Anthropic image source (base64 + HTTP URL paths); tool calls remapped
aws-bedrock (keyless / IAM)	convertOpenAIToBedrockMessages + ConverseCommand (cloud-auth.ts:215)	Uses the AWS SDK Converse API directly; bidirectional conversion includes system, messages, toolConfig, inferenceConfig; response converted back to OpenAI shape via convertBedrockToOpenAIResponse
aws-bedrock (API key)	OpenAI-compatible passthrough	When the deployment uses an API key rather than IAM
google-vertex	convertOpenAIToVertexNative ↔ convertVertexNativeToOpenAI (vertex-ai-helpers.ts)	System messages → systemInstruction; assistant role → model; image / text parts mapped; generationConfig reconstructed; streaming chunks re-shaped per-chunk
openai-azure / azure	Endpoint rewrite (deployment-aware)	Wire format preserved (Azure speaks OpenAI)
google (Gemini direct)	OpenAI-compatible passthrough	Where the deployment uses the OpenAI-compatible Gemini endpoint
groq / together / openrouter / mistral / fireworks / deepseek	None (pass-through)	All speak OpenAI-compatible
cohere	Routed via the /v2/chat adapter for native; or OpenAI-compatible for the Cohere compatibility endpoint	Two paths supported
ollama	OpenAI-compatible REST	Native pass-through
custom	OpenAI-compatible	Any OpenAI-spec endpoint

Cross-protocol translation matrix (inbound × outbound — what’s actually wired):

Inbound SDK	OpenAI	Anthropic	Bedrock	Vertex AI	Azure	Cohere	Groq / Together / OpenRouter / Mistral / Fireworks / DeepSeek	Ollama / vLLM / TGI / Triton (custom)
OpenAI SDK (/v1/chat/completions)	✅	✅	✅	✅	✅	✅	✅	✅
OpenAI SDK (/v1/completions, legacy)	✅	✅ (via chat-completion shim)	✅	✅	✅	✅	✅	✅
OpenAI embeddings (/v1/embeddings)	✅	n/a	✅ Bedrock Titan / Cohere via SDK	✅ Vertex embeddings	✅ Azure embeddings	✅	✅	✅
OpenAI Images / Audio	✅	n/a	n/a (use Bedrock image models via OpenAI shape)	✅ Imagen via shape	✅ Azure DALL-E	n/a	varies by provider	n/a
Anthropic SDK (/v1/messages)	passthrough-only (v2.0.105)	✅	passthrough-only	passthrough-only	passthrough-only	passthrough-only	passthrough-only	passthrough-only
Vertex AI SDK (/v1beta/models/…)	✅	✅	✅	✅	✅	✅	✅	✅
Azure OpenAI SDK (/openai/deployments/…)	✅	✅	✅	✅	✅	✅	✅	✅
Cohere V2 SDK (/v2/chat)	✅	✅	✅	✅	✅	✅	✅	✅
Bedrock SDK (raw Converse API)	❌ not currently an inbound endpoint — use the OpenAI SDK pointed at AgentWatch instead	❌	❌	❌	❌	❌	❌	❌

Legend:

• ✅ — Inbound format is accepted; gateway translates request + response and routes to the configured upstream provider; full DLP / budget / audit / circuit-breaker pipeline applies.
• passthrough-only — Inbound endpoint exists but is currently routed natively to the matching upstream protocol; cross-translation through this endpoint is not implemented (use a different inbound endpoint instead).
• ❌ — Not an inbound endpoint in v2.0.105.

Concrete examples this matrix unlocks:

App is written in…	Upstream configured in AgentWatch is…	What happens
OpenAI Python SDK	Anthropic Claude only	Works — request body translated to Anthropic Messages, response translated back to OpenAI Chat-Completion shape
OpenAI Node SDK	AWS Bedrock (IAM keyless)	Works — uses SDK ConverseCommand under the hood; response re-shaped to OpenAI
OpenAI Python SDK	Google Vertex AI (Gemini)	Works — messages → contents, system → systemInstruction, assistant → model
Cohere SDK	Anthropic Claude	Works — Cohere request → OpenAI shape → Anthropic translation; response → OpenAI shape → Cohere V2 shape
Azure OpenAI Node SDK	OpenAI (real) or Mistral or DeepSeek	Works — Azure deployment URL is rewritten to the configured upstream
Vertex AI Python SDK	OpenAI GPT-4	Works — Vertex request shape converted to OpenAI
Anthropic Python SDK on /v1/messages	Anthropic	Works — native passthrough
Anthropic Python SDK on /v1/messages	OpenAI	Not supported in v2.0.105 — /v1/messages is a native Anthropic passthrough endpoint. Workaround: use the OpenAI SDK with the same upstream (Anthropic adapter handles the reverse).
AWS Bedrock SDK (raw Converse API)	anything	Not supported in v2.0.105 — no inbound Converse endpoint. Switch the app to the OpenAI SDK pointed at the AgentWatch base URL; the gateway will route to Bedrock for you.

Practical implication: customers can swap upstream providers (e.g., during an OpenAI outage, or for cost optimization, or to comply with a regional sovereignty requirement) without changing a line of application code — provided the inbound SDK is OpenAI, Vertex AI, Azure OpenAI, or Cohere V2. The Anthropic SDK /v1/messages and the Bedrock Converse API as inbound formats are documented gaps for the next release.

---