OpenAI Compatible API Coverage

Full: Chat Completions, Completions, Embeddings, Models, Images (gen/edit/var), Audio (transcribe/translate/speech), Batches, Reranking, Fine-tuning (jobs/events/checkpoints), Assistants, Threads, Messages, Runs (+steps+tools), Files, Moderations, Vector Stores (+file batches). Plus: Anthropic Messages, Anthropic Token Counting, Vertex AI, Azure OpenAI, Cohere V2.

38.1 Context handling and token counting (v2.0.238+)

POST /v1/messages/count_tokens — Anthropic-compatible token counting, so an application can measure a request instead of estimating it. A local estimate cannot see what the serving engine adds to a prompt: its own injected system prompt, and the tool schemas it templates into the context (which can run to several thousand tokens for a client with many tools). Estimating those is the usual cause of a request that unexpectedly exceeds the model’s limit.

Request body is the same shape as /v1/messages (model, messages, optional system and tools). Response:

Field	Meaning
input_tokens	Total input size (the Anthropic-compatible field)
context_window	The model’s context window, or null if unknown
safety_margin	Headroom the gateway reserves
max_output_tokens_available	Room left for the response
granted_max_tokens	The max_tokens that would actually be granted
breakdown	Split by content, tools, scaffolding and images
fits	Whether the request fits at all

Automatic context fitting. Before forwarding, the gateway checks that the request fits the model’s context window and, if needed, reduces max_tokens so it does. It only ever lowers a requested value — never raises one, and never adds one that was not supplied. A shorter possible answer is preferable to a rejected request. When a request is close to the limit and the provider supports token counting, the gateway asks the provider for an exact count rather than relying on its own estimate.

Over-long requests fail fast. A request that cannot fit under any output budget returns a 400 with code CONTEXT_LENGTH_EXCEEDED and an explanation. This is deliberate: such a request is deterministic, so retrying it unchanged can never succeed. Genuine upstream faults remain retryable.

---