Amazon EKS Install Guide

Dream big, build fast. This guide installs the AgentWatch gateway into an existing Amazon EKS cluster using the Helm chart shipped in the product repo (helm/agentwatch), fronted by an AWS Application Load Balancer with an ACM certificate and backed by an EBS gp3 volume.

It is a do-this-then-this runbook. Every command is copy-paste ready; replace the values in the Fill these in first table and nothing else needs editing.

Scope. This guide covers the server (the gateway + admin console + Usage Insights). Getting employee laptops and browsers to route through it is a separate step — see the Install & Setup Guide and Endpoint Enrollment Guide. Deploying the optional AgentFleet agent into the same cluster is covered in §13.

---

Fill these in first

Variable	Example	What it is
CLUSTER	prod-use1	Your EKS cluster name
REGION	us-east-1	AWS region of the cluster
ACCOUNT_ID	123456789012	AWS account ID
NS	agentwatch	Kubernetes namespace to install into
RELEASE	aw	Helm release name (resource names derive from it)
HOSTNAME	agentwatch.acme.com	Public DNS name users and API clients will hit
ACM_ARN	arn:aws:acm:us-east-1:…:certificate/…	ACM certificate covering HOSTNAME
ADMIN_EMAIL	admin@acme.com	Seeded super-admin login

export CLUSTER=prod-use1 REGION=us-east-1 ACCOUNT_ID=123456789012
export NS=agentwatch RELEASE=aw
export HOSTNAME=agentwatch.acme.com
export ACM_ARN=arn:aws:acm:us-east-1:123456789012:certificate/REPLACE
export ADMIN_EMAIL=admin@acme.com

---

What you end up with

            Route 53  agentwatch.acme.com
                   │  (A/ALIAS)
                   ▼
        ┌──────────────────────────┐
        │  AWS ALB  (HTTPS :443)   │   ACM cert, idle_timeout 300s
        │  ingressClassName: alb   │   health check → /health
        └───────────┬──────────────┘
                    │ target-type: ip
        ┌───────────▼──────────────┐   Namespace: agentwatch
        │ Service aw-agentwatch    │   ClusterIP :8787
        └───────────┬──────────────┘
        ┌───────────▼──────────────┐
        │ StatefulSet (1 replica)  │   iterateai/agentwatch:<version>
        │  uid/gid 1001, non-root  │   probes → /health
        └───┬──────────────────┬───┘
            │                  │
      PVC …-data 10Gi    PVC …-logs 5Gi     ← EBS gp3, WaitForFirstConsumer
      /app/data (SQLite) /app/logs           helm.sh/resource-policy: keep

One replica is correct, not a limitation to fix. The default datastore is SQLite (file:/app/data/agentwatch.db), which has a single writer. replicaCount must stay 1 unless you move to the external-database tier (§12.3).

---

Step 1 — Prerequisites

Tools on your workstation

Tool	Minimum	Check
kubectl	1.27	kubectl version --client
helm	3.12	helm version
aws CLI	v2	aws sts get-caller-identity
eksctl	0.180	eksctl version (only used for add-on IAM wiring)

Cluster

• An EKS cluster (1.27+) with at least one managed node group.
• Node sizing: the chart requests 500m CPU / 512Mi memory and limits to 2 CPU / 8Gi memory. Pick an instance with headroom above 8 GiB — m6i.xlarge (4 vCPU / 16 GiB) is the comfortable default for up to ~500 users. m6i.large (8 GiB) cannot hold the limit alongside system pods.
• Graviton / arm64: the image is published as a multi-arch manifest, but the arm64 leg is built best-effort and a given tag can ship amd64-only. Confirm before choosing m7g:

  docker buildx imagetools inspect iterateai/agentwatch:latest | grep -A1 linux/arm64

  No arm64 line ⇒ use an x86_64 node group for that tag.

Get the chart. The chart lives in the product repo; there is no public Helm registry.

git clone https://github.com/iterateai/agentwatch.git
cd agentwatch          # chart path used below: ./helm/agentwatch

Connect kubectl

aws eks update-kubeconfig --name "$CLUSTER" --region "$REGION"
kubectl get nodes

---

Step 2 — Cluster add-ons

Two add-ons are required. Without the first, the pod sits in Pending forever with an unbound PVC; without the second, the Ingress object is created but no load balancer ever appears.

2a. EBS CSI driver (required — persistent storage)

EKS 1.23+ has no in-tree EBS provisioner. Install the managed add-on with its own IAM role:

# IRSA role for the driver
eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa --namespace kube-system \
  --cluster "$CLUSTER" --region "$REGION" \
  --role-name AmazonEKS_EBS_CSI_DriverRole \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
  --role-only --approve

# the add-on itself
eksctl create addon --name aws-ebs-csi-driver \
  --cluster "$CLUSTER" --region "$REGION" \
  --service-account-role-arn arn:aws:iam::${ACCOUNT_ID}:role/AmazonEKS_EBS_CSI_DriverRole \
  --force

kubectl -n kube-system get pods -l app.kubernetes.io/name=aws-ebs-csi-driver

2b. AWS Load Balancer Controller (required — ALB ingress)

curl -o alb-iam-policy.json \
  https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.8.1/docs/install/iam_policy.json

aws iam create-policy --policy-name AWSLoadBalancerControllerIAMPolicy \
  --policy-document file://alb-iam-policy.json 2>/dev/null || true

eksctl create iamserviceaccount \
  --cluster "$CLUSTER" --region "$REGION" \
  --namespace kube-system --name aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

helm repo add eks https://aws.github.io/eks-charts && helm repo update
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  -n kube-system \
  --set clusterName="$CLUSTER" \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set region="$REGION"

kubectl -n kube-system rollout status deploy/aws-load-balancer-controller

Subnet tags. The controller discovers subnets by tag. Public subnets need kubernetes.io/role/elb=1; private subnets need kubernetes.io/role/internal-elb=1. Clusters created by eksctl get these automatically; hand-built VPCs frequently do not — this is the #1 cause of “Ingress created, no ALB”.

2c. Optional add-ons

Add-on	Why you’d add it
ExternalDNS	Creates the Route 53 record from the Ingress automatically (otherwise §7 is manual)
External Secrets Operator	Sources AgentWatch secrets from AWS Secrets Manager instead of --set (§4, option C)
VolumeSnapshot CRDs + EBS snapshot class	Scheduled point-in-time snapshots of the data volume (§11)
Metrics Server	kubectl top pod for right-sizing memory

---

Step 3 — Namespace and gp3 StorageClass

EKS ships a gp2 default StorageClass with volumeBindingMode: Immediate. Create a gp3 class with WaitForFirstConsumer — this is what keeps the volume from being provisioned in an Availability Zone the pod can’t schedule into.

kubectl create namespace "$NS"

cat <<'EOF' | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
parameters:
  type: gp3
  encrypted: "true"
  fsType: ext4
provisioner: ebs.csi.aws.com
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
EOF

reclaimPolicy: Retain means deleting the PVC does not delete your data — deliberate, and consistent with the chart’s helm.sh/resource-policy: keep annotation on both PVCs.

---

Step 4 — Secrets

AgentWatch refuses to start with an empty JWT_SECRET — there is no weak default. Four values matter:

Key	Required	Consequence of not setting it
JWT_SECRET	Yes	Pod exits at boot
SUPER_ADMIN_PASSWORD	Strongly recommended	No way to log in as the seeded super-admin
ENCRYPTION_KEY	Production: yes	Provider keys and tenant secrets are encrypted with a public built-in default
BODY_ARCHIVE_KEY_HEX, CHAT_ENCRYPTION_KEY	Production: yes	Archived bodies / chat files encrypted with public defaults

Set the encryption keys before first boot. Changing them later does not re-encrypt existing data.

Generate them:

export JWT_SECRET=$(openssl rand -hex 32)
export ENCRYPTION_KEY=$(openssl rand -hex 64)
export BODY_ARCHIVE_KEY_HEX=$(openssl rand -hex 32)
export CHAT_ENCRYPTION_KEY=$(openssl rand -hex 32)
export SUPER_ADMIN_PASSWORD='<a strong password>'

Pick one delivery option:

Option A — chart-managed Secret (simplest). Pass values at install time with --set (§6). Nothing is committed; the chart renders a Secret named <release>-agentwatch-secrets.

Option B — pre-created Secret you manage.

kubectl -n "$NS" create secret generic agentwatch-secrets \
  --from-literal=JWT_SECRET="$JWT_SECRET" \
  --from-literal=SUPER_ADMIN_PASSWORD="$SUPER_ADMIN_PASSWORD" \
  --from-literal=ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  --from-literal=BODY_ARCHIVE_KEY_HEX="$BODY_ARCHIVE_KEY_HEX" \
  --from-literal=CHAT_ENCRYPTION_KEY="$CHAT_ENCRYPTION_KEY"

then in your values file set secrets.create: false and secrets.existingSecret: agentwatch-secrets.

Option C — AWS Secrets Manager via External Secrets Operator. Store the same five keys in one Secrets Manager secret and render it into the namespace with an ExternalSecret; then point secrets.existingSecret at the generated Kubernetes Secret name. Use this when a human should never hold the values.

Provider API keys (OpenAI, Anthropic, Bedrock, …) are not set here. They are entered in the admin console after first login and stored encrypted in the database.

---

Step 5 — The EKS values overlay

Save as values-eks.yaml. Every EKS-specific decision is in this one file.

# values-eks.yaml — AgentWatch on Amazon EKS
replicaCount: 1                      # SQLite: single writer. Do not raise.

image:
  repository: iterateai/agentwatch
  tag: "2.0.251"                     # pin a version in production; "latest" for a pilot
  pullPolicy: IfNotPresent           # use Always only with tag: latest

service:
  type: ClusterIP                    # the ALB targets pod IPs, not a NodePort
  port: 8787
  targetPort: 8787

ingress:
  enabled: true
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing          # internal = private ALB
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80},{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: "443"
    alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:us-east-1:123456789012:certificate/REPLACE"
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/healthcheck-interval-seconds: "15"
    alb.ingress.kubernetes.io/success-codes: "200"
    # CRITICAL for LLM traffic: the ALB default idle timeout is 60s. Large
    # prompts routinely take 30-90s before the first byte, and streaming
    # responses run longer — 60s silently truncates them into 504s.
    alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=300
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
    # Uncomment if ExternalDNS is installed:
    # external-dns.alpha.kubernetes.io/hostname: agentwatch.acme.com
  hosts:
    - host: agentwatch.acme.com
      paths:
        - path: /
          pathType: Prefix
  tls: []                            # TLS terminates at the ALB via certificate-arn

env:
  NODE_ENV: production
  PORT: "8787"
  DATABASE_URL: file:/app/data/agentwatch.db
  ENABLE_PATH_PREFIX_ROUTING: "true"
  EVENT_PIPELINE_KAFKA_ENABLED: "false"
  EVENT_PIPELINE_FILE_ARCHIVER_ENABLED: "true"
  # Upstream read timeout. Keep it BELOW the ALB idle timeout above so the app
  # returns a clean 504 instead of the LB severing the connection.
  AGENTWATCH_FETCH_TIMEOUT_MS: "120000"
  AGENTWATCH_SQLITE_INTEGRITY: "skip"
  SUPER_ADMIN_EMAIL: "admin@acme.com"
  # Per-IP DoS backstop. Behind an ALB, many clients share few source IPs —
  # leave this high and do real limiting per tenant/key in the console.
  AW_IP_RATE_MAX: "5000"

secrets:
  create: true                       # Option A; values supplied via --set at install
  jwtSecret: ""
  superAdminPassword: ""
  encryptionKey: ""
  bodyArchiveKeyHex: ""
  chatEncryptionKey: ""

persistence:
  data:
    enabled: true
    size: 20Gi                       # see §12.1 for sizing
    storageClass: gp3
    accessMode: ReadWriteOnce
    mountPath: /app/data
  logs:
    enabled: true
    size: 10Gi
    storageClass: gp3
    accessMode: ReadWriteOnce
    mountPath: /app/logs

resources:
  limits:   { cpu: "2", memory: 8Gi }
  requests: { cpu: "500m", memory: 1Gi }

# Do NOT remove fsGroup: a freshly provisioned EBS volume is root-owned, and
# fsGroup is what makes it writable by the uid-1001 app user.
podSecurityContext:
  runAsNonRoot: true
  fsGroup: 1001

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop: ["ALL"]
    add:  ["NET_BIND_SERVICE"]

# EBS volumes are zonal. Pin the pod to one AZ so a reschedule can always
# re-attach its volume. Match this to a node group that lives in that AZ.
nodeSelector:
  topology.kubernetes.io/zone: us-east-1a

updateStrategy:
  type: RollingUpdate

dns:
  enabled: false                     # see §15 — run the DNS redirector outside the cluster

Replace the three placeholders: certificate-arn, host, SUPER_ADMIN_EMAIL, and the nodeSelector zone.

Why these annotations

Setting	Reason
target-type: ip	The ALB talks to pod IPs directly — required for a ClusterIP Service, and it survives node churn
idle_timeout = 300	The single most common EKS failure. The 60s default kills long LLM calls and SSE streams mid-response
healthcheck-path: /health	Same endpoint the container’s own probes use
ssl-redirect: "443"	Forces HTTPS; the :80 listener only issues a redirect
AGENTWATCH_FETCH_TIMEOUT_MS < idle_timeout	The app owns the timeout, so clients get a structured error instead of a truncated stream

If your app is served under a sub-path (e.g. https://portal.acme.com/agentwatch), set env.BASE_PATH: "/agentwatch" and change the ingress path to match. The app strips the prefix itself — no rewrite-target annotation, and the health probes follow BASE_PATH automatically.

---

Step 6 — Install

helm install "$RELEASE" ./helm/agentwatch -n "$NS" -f values-eks.yaml \
  --set secrets.jwtSecret="$JWT_SECRET" \
  --set secrets.superAdminPassword="$SUPER_ADMIN_PASSWORD" \
  --set secrets.encryptionKey="$ENCRYPTION_KEY" \
  --set secrets.bodyArchiveKeyHex="$BODY_ARCHIVE_KEY_HEX" \
  --set secrets.chatEncryptionKey="$CHAT_ENCRYPTION_KEY" \
  --wait --timeout 10m

Watch it come up — first boot runs migrations and can take a few minutes (the startup probe allows up to 5):

kubectl -n "$NS" get pods -w
kubectl -n "$NS" logs -f statefulset/${RELEASE}-agentwatch
kubectl -n "$NS" get pvc
kubectl -n "$NS" get ingress

Expected: one Running / 1/1 pod, two Bound PVCs, and an Ingress whose ADDRESS column fills in with an ALB DNS name after 2–4 minutes.

Run helm install --dry-run --debug first if you want to inspect the rendered manifests without touching the cluster.

---

Step 7 — DNS and TLS

The ACM certificate is already attached to the listener by the certificate-arn annotation. All that remains is pointing your hostname at the ALB.

ALB=$(kubectl -n "$NS" get ingress ${RELEASE}-agentwatch \
       -o jsonpath='{.status.loadBalancer.ingress[0].hostname}')
echo "$ALB"

Create a Route 53 A-record with Alias = true targeting $ALB (never a CNAME at a zone apex). With ExternalDNS installed, uncomment the external-dns.alpha.kubernetes.io/hostname annotation instead and skip this step.

Then:

curl -sS https://$HOSTNAME/health
# {"ok":true,"service":"company-llm-aggregator","version":"2.0.251", ... }

---

Step 8 — Verify

# 1. Health through the ALB
curl -sS https://$HOSTNAME/health | head -c 200

# 2. Super-admin login works
curl -sS -X POST https://$HOSTNAME/api/auth/login \
  -H 'content-type: application/json' \
  -d "{\"email\":\"$ADMIN_EMAIL\",\"password\":\"$SUPER_ADMIN_PASSWORD\"}" | head -c 120

# 3. Console loads
open https://$HOSTNAME

Then, in the console:

1. Add a provider (Settings → Providers) with a real API key, and confirm models sync.
2. Create a tenant/employee and copy the generated ak_… gateway key.
3. Send one governed request and confirm it lands in Usage Insights within seconds:

   curl -sS https://$HOSTNAME/v1/chat/completions \
     -H "authorization: Bearer ak_REPLACE" \
     -H 'content-type: application/json' \
     -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}]}'

4. Test DLP: enable DLP in Company Settings, send a message containing a fake SSN (123-45-6789), and confirm the row shows Redacted or Blocked — and that the stored preview shows a placeholder, never the raw value.

If step 3 works but step 4’s row never appears, the request is not reaching the gateway — check the ALB target group health before anything else.

---

Step 9 — Apply the recommended posture (optional)

The repo ships a Job that applies the recommended DLP / rate-limit posture over the admin API — the Kubernetes equivalent of the Ansible agentwatch_config role.

Edit two fields before applying helm/agentwatch/recommended-posture-job.yaml:

Field	Set to
AW_SVC	http://<release>-agentwatch:8787 (e.g. http://aw-agentwatch:8787)
secretKeyRef.name	<release>-agentwatch-secrets — the chart appends -secrets; the file’s shipped default omits it and the Job will fail to start without this edit

kubectl -n "$NS" apply -f helm/agentwatch/recommended-posture-job.yaml
kubectl -n "$NS" logs job/agentwatch-posture -f

It is idempotent (it read-merges current settings). Flip between governance (DLP_MODE=monitor, actions log) and preventive (DLP_MODE=block, PII_ACTION=redact) by editing the env block, deleting the old Job, and re-applying.

---

Step 10 — Upgrades and rollback

Upgrades are a plain image swap. Both PVCs are retained, so data survives.

# Pinned tag (recommended)
helm upgrade "$RELEASE" ./helm/agentwatch -n "$NS" -f values-eks.yaml \
  --set image.tag=2.0.252 --reuse-values --wait --timeout 10m

# If image.tag is "latest", force a re-pull:
kubectl -n "$NS" rollout restart statefulset/${RELEASE}-agentwatch
kubectl -n "$NS" rollout status  statefulset/${RELEASE}-agentwatch

Expect a brief gap. One replica on a ReadWriteOnce volume means the old pod must fully terminate and detach the EBS volume before the new one attaches — typically 30–90 seconds. Schedule it, or run the maintenance banner.

helm history "$RELEASE" -n "$NS"
helm rollback "$RELEASE" <REVISION> -n "$NS"

Roll back the image, not the data. Take a snapshot (§11) before any upgrade you might reverse.

---

Step 11 — Backup and disaster recovery

What must be backed up: the …-data PVC. It holds the SQLite database, the encrypted body archive, chat files, and vector data. The …-logs PVC is disposable.

Continuous — EBS snapshots. Install the external-snapshotter CRDs plus a VolumeSnapshotClass for ebs.csi.aws.com, then:

cat <<EOF | kubectl apply -f -
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: agentwatch-data-$(date +%Y%m%d)
  namespace: $NS
spec:
  volumeSnapshotClassName: ebs-csi-snapclass
  source:
    persistentVolumeClaimName: ${RELEASE}-agentwatch-data
EOF

Or attach the underlying EBS volume to AWS Backup with a daily plan — simplest for compliance reporting.

Consistent SQLite copy (hot, no downtime). A raw file copy of a live SQLite DB can be torn; use the online backup API:

POD=$(kubectl -n "$NS" get pod -l app.kubernetes.io/instance=$RELEASE -o name | head -1)
kubectl -n "$NS" exec "$POD" -- \
  sqlite3 /app/data/agentwatch.db ".backup '/tmp/aw-backup.db'"
kubectl -n "$NS" cp "${POD#pod/}:/tmp/aw-backup.db" ./aw-backup-$(date +%F).db

Also back up your encryption keys. An …-data snapshot without ENCRYPTION_KEY, BODY_ARCHIVE_KEY_HEX, and CHAT_ENCRYPTION_KEY is unrecoverable. Keep them in Secrets Manager.

Restore: create a PVC from the snapshot named ${RELEASE}-agentwatch-data, then helm install with the same release name — the chart adopts an existing PVC of that name.

---

Step 12 — Sizing, scaling, and the database tier

12.1 Storage

Users	data PVC	logs PVC	Notes
100	20 Gi	10 Gi	Comfortable for a year
500	50 Gi	20 Gi	Watch the encrypted body archive under /app/data/{org}/bodies/
1,000+	100 Gi+	20 Gi	Consider the external-DB tier (12.3)

gp3 supports online expansion (allowVolumeExpansion: true): edit the PVC’s spec.resources.requests.storage and the filesystem grows without a restart.

12.2 Memory

The process holds a modest JS heap but a large native footprint (SQLite native heap, vector buffers, HTTP/2 buffers). The 8Gi limit exists because a 4Gi limit produced OOM kills during SQL aggregation over multi-million-row usage tables. If you see OOMKilled, raise the limit rather than the replica count.

kubectl -n "$NS" top pod
kubectl -n "$NS" describe pod <pod> | grep -A3 'Last State'

12.3 Beyond one replica

More than one replica requires two changes, in this order:

1. External database. Point DATABASE_URL at Amazon RDS for PostgreSQL (see docs/POSTGRESQL_SETUP.md). SQLite cannot serve concurrent writers.
2. Shared Redis. With multiple pods behind one ALB, rate limits and shared state must live in ElastiCache for Redis, or each replica limits in isolation. Configure it in the console (Enterprise Guide §7.10).

Only then raise replicaCount, and drop the nodeSelector zone pin (with RDS + Redis there is no zonal volume to follow).

12.4 AZ pinning

EBS volumes are zonal. With WaitForFirstConsumer the volume is created in whatever AZ the pod first lands in — and thereafter the pod can only run in that AZ. The nodeSelector in §5 makes that explicit instead of accidental. If the AZ fails, restore from snapshot into another AZ (§11) rather than waiting for the volume.

---

Step 13 — Optional: AgentFleet agent in the same cluster

To attribute and govern in-cluster workload traffic (not just laptops), deploy the fleet agent as a DaemonSet — one agent per node — and point your app pods at it.

helm install agentwatch-fleet ./client-agent/deploy/helm/agentwatch-agent \
  -n agentwatch --create-namespace \
  --set agentwatch.serverUrl="https://$HOSTNAME" \
  --set agentwatch.email="agent-fleet@acme.com" \
  --set-string secret.apiKey="ak_REPLACE"

Then in each workload:

env:
  - name: NODE_IP
    valueFrom: { fieldRef: { fieldPath: status.hostIP } }
  - name: OPENAI_BASE_URL
    value: "http://$(NODE_IP):8088/v1"

The agent stamps node/pod/namespace into its 30-second heartbeat, which populates the Fleet view in the Observe workspace. For per-pod (rather than per-node) attribution, inject it as a sidecar instead — see client-agent/deploy/k8s/sidecar-injection.yaml. The agent fails open: if the gateway is unreachable, traffic passes through rather than breaking your workload.

---

Step 14 — Optional: A2A mTLS listener

Agent-to-agent mutual TLS runs on a separate port (8788) and requires a real client-certificate handshake, so it cannot go behind the ALB’s HTTPS listener — TLS termination there would destroy the client cert. Expose it with a network load balancer instead:

# add to values-eks.yaml
env:
  A2A_MTLS_ENABLED: "true"
  A2A_MTLS_PORT: "8788"

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: ${RELEASE}-agentwatch-mtls
  namespace: $NS
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internal
spec:
  type: LoadBalancer
  ports:
    - name: mtls
      port: 8788
      targetPort: 8788
      protocol: TCP
  selector:
    app.kubernetes.io/name: agentwatch
    app.kubernetes.io/instance: $RELEASE
EOF

Set secrets.encryptionKey before enabling this — it is what encrypts the per-org CA key at rest. Never rotate ENCRYPTION_KEY on a deployment with a live CA. Details in docs/A2A_mTLS.md.

---

Step 15 — Optional: the DNS redirector

The chart can render the DNS redirector in-cluster (dns.enabled=true), but the supported production pattern is a standalone container on a dedicated public IP that owns port 53. In EKS it needs a Service exposing UDP and TCP on privileged ports, and mixed UDP/TCP on a single load balancer is not supported by every AWS load balancer configuration. Run it outside the cluster unless you have a specific reason not to — see dns-server/README.md.

---

Step 16 — Troubleshooting

Symptom	Cause	Fix
Pod Pending, PVC Pending, event no persistent volumes available	EBS CSI driver missing or its IRSA role is wrong	§2a. Check kubectl -n kube-system logs deploy/ebs-csi-controller
Ingress exists, ADDRESS stays empty	LB controller not running, or subnets lack kubernetes.io/role/elb tags	§2b. kubectl -n kube-system logs deploy/aws-load-balancer-controller
CrashLoopBackOff, log says JWT secret required	JWT_SECRET empty	§4 — it has no default by design
ALB target group unhealthy, pod is Ready	Health check path doesn’t match BASE_PATH	Set healthcheck-path to <BASE_PATH>/health
Long prompts return 504 after ~60s	ALB idle timeout still at the default	Add idle_timeout.timeout_seconds=300 (§5)
Streaming responses truncate mid-answer	Same as above	Same as above
OOMKilled	Memory limit below the native footprint	Raise resources.limits.memory; do not add replicas (§12.2)
database is locked	More than one replica against SQLite	replicaCount: 1, or move to RDS (§12.3)
Pod won’t reschedule after a node dies	Volume is in another AZ	§12.4 — pin the zone, or restore from snapshot
First boot takes minutes	Migrations + model catalog sync	Normal; the startup probe allows 5 minutes
ImagePullBackOff on Graviton nodes	Tag has no arm64 manifest	§1 — verify with docker buildx imagetools inspect
Permission denied writing /app/data	fsGroup removed from the values file	Restore podSecurityContext.fsGroup (§5)

Useful one-liners:

kubectl -n "$NS" describe pod -l app.kubernetes.io/instance=$RELEASE | tail -30
kubectl -n "$NS" logs statefulset/${RELEASE}-agentwatch --tail=200
kubectl -n "$NS" get events --sort-by=.lastTimestamp | tail -20
kubectl -n "$NS" port-forward svc/${RELEASE}-agentwatch 8787:8787   # bypass the ALB

That last one is the fastest way to split the problem: if http://localhost:8787/health is healthy, the fault is in the ALB/DNS/TLS layer, not the app.

---

Step 17 — Uninstall

helm uninstall "$RELEASE" -n "$NS"

Your data is deliberately left behind. Both PVCs carry helm.sh/resource-policy: keep, and the gp3 class uses reclaimPolicy: Retain. A reinstall with the same release name reattaches them.

To destroy the data as well — irreversible:

kubectl -n "$NS" delete pvc ${RELEASE}-agentwatch-data ${RELEASE}-agentwatch-logs
kubectl delete namespace "$NS"
# Retained PVs and their EBS volumes must then be deleted by hand:
kubectl get pv | grep "$NS"

---

Appendix A — Production readiness checklist

• Image tag pinned to a version, not latest
• ENCRYPTION_KEY, BODY_ARCHIVE_KEY_HEX, CHAT_ENCRYPTION_KEY set before first boot and stored in Secrets Manager
• SUPER_ADMIN_PASSWORD rotated from any bootstrap value
• ALB idle_timeout ≥ 300s and above AGENTWATCH_FETCH_TIMEOUT_MS
• ACM certificate covers the hostname and auto-renews
• gp3 StorageClass with Retain + WaitForFirstConsumer
• Daily snapshot of the data PVC, restore rehearsed once
• Pod zone pinned and matched to a node group in that AZ
• DLP posture applied and verified with a fake-PII probe
• Alarm email sender configured (Settings → Email Notifications)
• kubectl top pod baseline captured for memory right-sizing
• ALB access logs enabled to S3 if required for audit

Appendix B — Rough AWS cost (single-replica tier)

Component	Example	Approx. monthly
Node capacity for the pod	m6i.xlarge share	$60–140
ALB	1 ALB + modest LCUs	$20–30
EBS gp3	30 Gi total	$3
Snapshots	daily, 30-day retention	$2–5
Route 53	hosted zone + queries	$1

Excludes LLM provider spend, which AgentWatch measures and attributes for you. EKS control-plane charges ($73/mo) are assumed already borne by the existing cluster.

Appendix C — Where to go deeper

Topic	Guide
First-run setup, enrollment, verification	AgentWatch_Install_Guide
Architecture, multi-tenancy, scale, Redis, operations	AgentWatch_Enterprise_Guide
Product reference for admins	AgentWatch_User_Guide
Gateway routes and tenant modes	AgentWatch_Gateway_Rules_Config
API integration with delegated identity	AgentWatch_Delegated_Identity
Endpoint / device enrollment	AgentWatch_Endpoint_Enrollment_Guide
SAML SSO	AgentWatch_SAML_SSO_Setup
Symptom-driven fixes	AgentWatch_Troubleshooting
PostgreSQL / external database	docs/POSTGRESQL_SETUP.md
A2A mutual TLS	docs/A2A_mTLS.md

_{© Iterate.ai · AgentWatch · Dream big, build fast.}