Caching

Semantic Cache: Vector-similarity response cache (LanceDB). Configurable similarity threshold (default 0.92), TTL-based expiration (default 3600s), max 10K entries, configurable embedding model. Returns the cached response for semantically similar queries without re-calling the upstream LLM — typical workload savings: 20–40% cost reduction on FAQ-style support, code-completion, and other repetitive prompt patterns.

Prompt Caching (provider-native passthrough): transparent passthrough of provider-native prompt caching — Anthropic cache_control breakpoints (documented as up to 90% cost reduction for cache hits on large system prompts and tool definitions) and OpenAI automatic prompt cache (with cached-token tracking surfaced in usage reports). Min 1024 tokens to qualify. Enabled by default, zero configuration required — applications get the savings automatically.

Response Cache: Request normalization with configurable TTL — exact-match cache for deterministic queries (e.g., classifier prompts with temperature=0).

---