Skip to content
Docs

AI Assistant Settings — System LLM

The System LLM is the AI model AgentWatch uses internally for:

  • AI Session Insights — RAG-powered chat that lets admins ask questions about session data (e.g., “What are the most common user queries this week?”)
  • Response Enrichment — LLM fallback tool used in Enriched routing rules
  • Gateway Model Provider — System intelligence for enriched routing rule pre/post processing

The System LLM is configured once at the platform level (and optionally overridden per client organization).

Supported Providers

Section titled “Supported Providers”
Provider Auth Method Notes
OpenAI API key Default. Works out of the box with a standard key.
Anthropic API key Uses native Messages API format.
AWS Bedrock IAM role or API key (proxy) Native SDK path for IAM; proxy path for custom gateways.
Google Vertex AI Service account JSON or OAuth token Auto-refreshes tokens from service account JSON.
Google AI (Gemini) API key OpenAI-compatible Gemini endpoint.
Azure OpenAI API key Requires your Azure resource base URL.
Groq API key OpenAI-compatible.
Ollama / Local LLM Optional Point at your local Ollama or vLLM server.
Other OpenAI-compatible API key Any server exposing /v1/chat/completions.

Step-by-Step: Configuring the System LLM

Section titled “Step-by-Step: Configuring the System LLM”

Step 1: Open AI Assistant Settings

  1. Log in to the AgentWatch Admin Panel as super_admin
  2. Go to Admin → Settings → AI Assistant

Step 2: Enable the System LLM

  • Check Enable System LLM

Step 3: Select Provider and Model

  • Choose your provider from the Provider dropdown
  • Click the ↻ refresh icon next to the model dropdown to fetch the live model list from your provider. For AWS Bedrock, this calls ListFoundationModels via the AWS SDK and returns only models enabled for on-demand inference in your account. For other providers it calls their /v1/models endpoint. A curated static fallback list is used if the live call fails.

Step 4: Enter Credentials

  • Fill in the API Key and (where required) the API Base URL — see provider-specific instructions below.
  • AWS Bedrock: The API Key field is hidden by default (IAM auth requires no key). Check “Using a Bedrock-compatible gateway proxy” to reveal the API Key field if you are routing through a proxy like LiteLLM.

Step 5: Test the Connection

  • Click Test Connection to verify credentials before saving.
  • The result appears inline next to the button:
    • ✓ Green — connected successfully
    • ✗ Red — descriptive error (e.g. “AWS credentials not found”, “Invalid API key”, “Model not found in this region”)
  • This sends a minimal "Hi" message (max 10 tokens) to confirm end-to-end connectivity.

Step 6: Save

  • Click Save System LLM
  • The status indicator in the top-right corner of the panel will show Connected (provider-name) when the configuration is valid.

Provider-Specific Configuration

Section titled “Provider-Specific Configuration”

OpenAI

Section titled “OpenAI”
Field Value
Provider OpenAI
Model gpt-4.1 (recommended), gpt-4o, o3
API Key Your OpenAI key from platform.openai.com
API Base URL (leave empty — uses api.openai.com automatically)

Anthropic

Section titled “Anthropic”
Field Value
Provider Anthropic
Model claude-sonnet-4-6 (recommended), claude-opus-4-6
API Key Your Anthropic key from console.anthropic.com
API Base URL (leave empty)

AWS Bedrock

Section titled “AWS Bedrock”

AWS Bedrock supports two authentication paths. Choose the one that matches your deployment.

Option A — Native IAM Authentication (Recommended for self-hosted / EC2 / ECS)

Section titled “Option A — Native IAM Authentication (Recommended for self-hosted / EC2 / ECS)”

AgentWatch calls the Bedrock Converse API directly using the AWS SDK default credential chain (EC2 instance role, ECS task role, environment variables, or ~/.aws/credentials). No API key is needed.

Field Value
Provider AWS Bedrock
Model anthropic.claude-sonnet-4-20250514-v1:0 (recommended) — see model table below
API Key (leave empty — IAM role handles authentication)
API Base URL https://bedrock-runtime.us-east-1.amazonaws.com (replace us-east-1 with your region)

AWS credential setup (choose one):

Environment variables (most common for containers):

Terminal window
export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=us-east-1

EC2 / ECS instance role: Attach an IAM role with the following policy to your instance/task — no environment variables needed:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeModel",
        "bedrock:InvokeModelWithResponseStream"
      ],
      "Resource": "arn:aws:bedrock:us-east-1::foundation-model/*"
    }
  ]
}

AWS credentials file (~/.aws/credentials):

[default]
aws_access_key_id     = AKIA...
aws_secret_access_key = ...
region                = us-east-1

Important: The API key field in the System LLM settings is not used for IAM authentication. It is only used when routing through a Bedrock proxy (Option B below). Leave it empty for native IAM.

Option B — Bedrock via OpenAI-Compatible Proxy (e.g. LiteLLM)

Section titled “Option B — Bedrock via OpenAI-Compatible Proxy (e.g. LiteLLM)”

If you run a proxy in front of Bedrock that exposes an OpenAI-compatible /v1/chat/completions endpoint:

Field Value
Provider AWS Bedrock
Model Your proxy’s model identifier
API Key Bearer token required by your proxy
API Base URL Your proxy’s base URL (e.g. http://litellm.internal:4000)

AgentWatch detects which path to use based on the API Base URL:

  • URL contains amazonaws.com → native SDK path (IAM, no key required)
  • Any other URL → proxy path (OpenAI-compatible HTTP, key used as Bearer token)

Available Bedrock Models

Section titled “Available Bedrock Models”
Model ID Display Name
anthropic.claude-sonnet-4-20250514-v1:0 Claude Sonnet 4 (Recommended)
anthropic.claude-haiku-4-5-20251001-v1:0 Claude Haiku 4.5 (Faster)
anthropic.claude-opus-4-20250514-v1:0 Claude Opus 4
anthropic.claude-3-5-sonnet-20241022-v2:0 Claude 3.5 Sonnet v2
anthropic.claude-3-5-haiku-20241022-v1:0 Claude 3.5 Haiku
amazon.nova-pro-v1:0 Amazon Nova Pro
amazon.nova-lite-v1:0 Amazon Nova Lite
meta.llama3-70b-instruct-v1:0 Llama 3 70B Instruct
amazon.titan-text-express-v1 Amazon Titan Text Express

Google Vertex AI

Section titled “Google Vertex AI”
Field Value
Provider Google Vertex AI
Model gemini-2.5-flash (recommended), gemini-2.5-pro
API Key Paste your Service Account JSON (recommended — AgentWatch auto-refreshes tokens), or a short-lived token from gcloud auth print-access-token
API Base URL https://REGION-aiplatform.googleapis.com/v1beta1/projects/PROJECT_ID/locations/REGION

Azure OpenAI

Section titled “Azure OpenAI”
Field Value
Provider Azure OpenAI
Model Your deployment name (e.g. gpt-4o)
API Key Your Azure OpenAI key from the Azure Portal
API Base URL https://YOUR-RESOURCE.openai.azure.com

Ollama (Local LLM)

Section titled “Ollama (Local LLM)”
Field Value
Provider Ollama
Model Click ↻ to load models from your Ollama instance
API Key (leave empty unless your server requires auth)
API Base URL http://localhost:11434 (default)

Troubleshooting

Section titled “Troubleshooting”
Symptom Likely Cause Fix
Test Connection: “AWS credentials not found” No IAM credentials configured on the server Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION env vars, or attach an EC2/ECS IAM role.
Test Connection: “Access denied. Ensure the IAM role has bedrock:InvokeModel permission” IAM role exists but lacks Bedrock permissions Attach bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream to the IAM policy.
Test Connection: “Model not found in this region” Model ID is correct but not enabled in the selected region Go to AWS Console → Bedrock → Model access and enable the model. Also verify the regional endpoint URL matches your region.
Test Connection: “Invalid OpenAI API key” Wrong or expired API key Regenerate your key at platform.openai.com.
Test Connection: “API key required for Bedrock gateway proxy mode” API Base URL is not an amazonaws.com URL but no API key was given Either clear the URL (to use native IAM) or check “Using a Bedrock-compatible gateway proxy” and enter your proxy’s bearer token.
Test Connection: “Could not reach … server is running” Ollama / local LLM not running or wrong port Verify the server is running: curl http://localhost:11434/v1/models
↻ Model refresh shows only static list AWS credentials not configured (live ListFoundationModels failed) Configure IAM credentials first, then click ↻ again to load the live model list.
“Connected (aws-bedrock)” shown but AI Session Insights returns “No LLM response received”** AWS credentials not configured on the server Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION environment variables, or attach an IAM role to the host.
“System LLM API key is not configured” The API key field is empty and the provider is not using native IAM For non-IAM providers, enter your API key. For native Bedrock IAM, leave the key empty and ensure AWS credentials are available via environment or instance role.
Anthropic “Invalid API Key” Wrong key format Anthropic keys start with sk-ant-. Verify in console.anthropic.com.
Vertex AI 403 / model not found Model not enabled in your GCP project Enable the model in Google Cloud Console → Vertex AI → Model Garden.

API Reference

Section titled “API Reference”
Method Endpoint Description
GET /admin/company/settings Read current System LLM configuration (key is masked)
POST /admin/company/settings Save System LLM settings (systemLlmEnabled, systemLlmProvider, systemLlmModel, systemLlmApiKey, systemLlmBaseUrl)
POST /admin/system-llm/models Fetch available models for a provider (live API call or static fallback). Body: { provider, baseUrl?, apiKey? }
POST /admin/system-llm/test Test provider connectivity. Sends a minimal message and returns { success, message? } on success or { success: false, error } with a human-readable error. Body: { provider, model, baseUrl?, apiKey? }