Semantic Caching (LanceDB)

What Is Semantic Caching?

Semantic Caching is AgentWatch’s intelligent response cache. Unlike a traditional exact-match cache (where "What is 2+2?" and "What's 2 plus 2?" would be cache misses), the semantic cache uses vector embeddings to detect when two prompts are asking essentially the same question — even when phrased differently. When a match is found above the similarity threshold, the cached response is returned instantly without calling the LLM, eliminating the token cost and latency entirely.

Example:

• First request: "Explain what TCP/IP is in simple terms" → LLM called, response cached
• Second request: "Can you describe TCP/IP in plain English?" → 0.95 similarity → cache hit, cached response returned immediately

How It Works

Incoming Prompt
       │
       ▼
┌───────────────────────────────────────┐
│  Generate embedding vector            │
│  (text-embedding-3-small, 1536 dims)  │
└──────────────┬────────────────────────┘
               │
               ▼
┌───────────────────────────────────────┐
│  Vector similarity search in LanceDB  │
│  Scoped to: same model + same tenant  │
│  Filter: expiresAt > now              │
└──────────────┬────────────────────────┘
               │
       ┌───────┴──────────┐
       ▼                  ▼
  similarity          similarity
  ≥ threshold         < threshold
  (cache hit)         (cache miss)
       │                  │
       ▼                  ▼
  Return cached      Call LLM → store
  response           response in LanceDB
  (no LLM cost)      for future queries

LanceDB — The Vector Store

AgentWatch uses LanceDB as the embedded vector database for semantic caching. LanceDB:

• Runs in-process alongside AgentWatch — no separate database server required
• Stores compressed embedding vectors on local disk in the configured storagePath
• Supports ANN (Approximate Nearest Neighbour) search with sub-millisecond query times on millions of vectors
• Is tenant-isolated — every search is filtered to the requesting tenant’s ID

For AI Insights (§7.13 / §12.10), a second LanceDB table stores embeddings of session logs, enabling natural-language questions over usage history (RAG).

Configuration Settings

Navigate to: Settings → AI Assistant → Semantic Caching

Setting	Description	Default
Enable Semantic Caching	Master on/off switch	Off
Similarity Threshold	0.5–1.0. Below this score = miss. Higher = more conservative caching	0.92
TTL (seconds)	How long a cached entry lives before expiry	3600 (1 hour)
Max Entries	Maximum cached prompts before oldest are evicted	10,000
Embedding Model	Model used to generate vectors	text-embedding-3-small
Storage Path	Local directory for LanceDB files	./data/semantic-cache

Note: Semantic caching requires the System LLM (§7.13) to be configured. The embedding model must be accessible via the same API key.

Similarity Threshold Guide

Threshold	Behaviour	Recommended for
0.98–1.0	Near-exact matches only	FAQs, fixed-form queries
0.92–0.97	Semantically equivalent phrasing	General enterprise use (default)
0.85–0.91	Broadly similar topics	High-volume, cost-sensitive deployments
< 0.85	Aggressive — may return off-topic responses	Not recommended

Cache Entry Fields

Each entry stored in LanceDB contains:

Field	Description
id	UUID
embedding	1536-dimension float32 vector
model	LLM model the original response was from
tenantId	Scopes the entry to one tenant — prevents cross-tenant cache bleed
messagesJson	Original prompt messages
responseJson	Cached LLM response
createdAt	Creation timestamp
expiresAt	Expiry timestamp (TTL applied)
hitCount	Number of times this entry served a cache hit
lastHit	Timestamp of the most recent hit

Cache Hit Response

When a semantic cache hit occurs, the response includes two additional fields:

{
  "id": "chatcmpl-cached-...",
  "choices": [ { "message": { "content": "..." } } ],
  "_cached": true,
  "_cache_similarity": 0.9634
}

Use _cached: true in your analytics or logging to distinguish cached from live responses.

Performance and Cost Impact

• Latency: Cache hits typically return in < 5 ms vs. 500–3000 ms for a live LLM call.
• Token savings: 100% of input and output tokens are saved on a cache hit.
• Hit rate: Typically 15–40% for knowledge-base or support use cases; lower for highly varied creative or code generation prompts.

Step-by-Step: Enabling Semantic Caching

1. Ensure the System LLM is configured (Settings → AI Assistant → System LLM).
2. Navigate to Settings → AI Assistant → scroll to Semantic Caching (LanceDB).
3. Toggle Enable Semantic Caching on.
4. Adjust the Similarity Threshold (start at 0.92 and tune based on observed hit/miss rates).
5. Set TTL based on how frequently your content changes (1 hour for dynamic data, 24 hours for stable knowledge bases).
6. Click Save Semantic Cache Settings.
7. Monitor the Status, Hits, Misses, and Hit Rate counters that appear below the save button.

Step-by-Step: Tuning the Cache

1. After one week of traffic, navigate to Settings → AI Assistant → Semantic Caching.
2. Check the Hit Rate counter. If it is below 10%, the threshold may be too high — try lowering by 0.02.
3. If users report receiving incorrect cached responses, the threshold is too low — raise it by 0.02.
4. To clear the cache (e.g., after a knowledge base update), delete the contents of the Storage Path directory and restart AgentWatch.

API Reference

Method	Endpoint	Description
GET	/api/admin/company/settings	Read semantic cache settings (in semanticCache* fields)
PUT	/api/admin/company/settings	Update semantic cache settings
GET	/api/admin/semantic-cache/stats	Live hit/miss/entry counts
DELETE	/api/admin/semantic-cache	Flush the entire semantic cache

---