Troubleshooting

Common install + runtime issues and their fixes. Tiphys's chat is the fastest troubleshooting surface for most of these — the agent already knows how to diagnose its own errors — but the common symptoms are collected here as a pre-chat reference.

Install

Pod stuck in Pending after helm install

• Most common cause: the ServiceAccount annotation doesn't match a real IAM role. Tiphys won't start if IRSA is misconfigured in production mode. Fix:

  kubectl annotate sa -n tiphys tiphys \
    eks.amazonaws.com/role-arn=arn:aws:iam::...:role/tiphys-irsa --overwrite
  kubectl rollout restart -n tiphys deploy/tiphys
  

• Less common: resource requests don't fit the node. Check kubectl describe pod -n tiphys — the scheduler's FailedScheduling event names the exact constraint that failed.

• No default StorageClass: the state volume never binds. Since v0.5.0 helm install stops with "This cluster has no default StorageClass" instead of leaving the pod in Pending. Pick a class from kubectl get storageclass and set persistence.storageClass, mark one as the default, or install a provisioner such as local-path-provisioner. (persistence.mode=configmap needs no volume, but then the admin password has to come from auth.builtin.existingSecret.)

Pod in CrashLoopBackOff

kubectl logs -n tiphys deploy/tiphys -c backend

• pydantic_core._pydantic_core.ValidationError on startup — a Helm value doesn't match the expected type. The traceback names the field.
• NoCredentialsError from botocore — IRSA isn't wired. Verify the SA annotation (above) + the role's trust policy (see Install Tiphys).

Chat returns errors

error_kind: permission_denied in chat

The agent hit a 403 — K8s RBAC, Prometheus auth, or AWS IAM. Tiphys surfaces the exact fix inline. Example:

error_kind: permission_denied
The agent was denied 'list' on Secret (namespace 'prod') — forbidden.

Fix:
Add this rule to the reader ClusterRole (...):
- apiGroups: [""]
  resources:
    - secrets
  verbs: ['get', 'list', 'watch']

After editing the chart, re-run `bash scripts/dev-deploy.sh` (or the
equivalent helm upgrade) to roll the new RBAC into the cluster.

The Fix: block is paste-ready — apply it to the policy named in the hint. Same shape for:

• K8s RBAC → paste into your customer-side fork of the reader ClusterRole (the rules are inlined in Permissions).
• Prometheus 401/403 → set integrations.prometheus.url to the unauthenticated in-cluster Service, or open a NetworkPolicy.
• AWS IAM → paste into the Cloud IAM policy (the JSON is inlined in IAM setup).

tool execution error: no scripted response for expr=...

You're running Tiphys without Prometheus configured (or auto-discovery didn't find it). Two fixes:

1. Install a kube-prometheus-stack release — Tiphys auto-detects Services whose name contains prometheus with port 9090.

2. Set the URL explicitly — if your Prometheus runs under a non-standard name:

   integrations:
     prometheus:
       url: http://my-prometheus.monitoring.svc:9090
   

Bedrock returns AccessDeniedException

Two distinct failure modes:

• IAM action missing — the IRSA role doesn't have bedrock:InvokeModel. Fix via Terraform or IAM console.
• Model access not requested — Bedrock in a fresh AWS account blocks every model by default until you opt in. Go to the Bedrock console → Model access → enable Claude Sonnet + Haiku in the target region.

The error message distinguishes them: "is not authorized" → IAM action missing; "You don't have access to the model" → model access not requested.

Reports + approvals

Reports view is empty

First check takes up to 30 minutes if the pod was restarted immediately after install (the first scheduled tick hasn't fired yet). Force one from the UI (Run now) or from chat ("run a health check now"). The endpoint behind both is POST /api/reports/run-now.

Approvals view is empty overnight

By default reports.recommenderMode=true — findings carry recommendations but the operator has to manually stage them. For "approvals populate overnight" behavior, set:

reports:
  recommenderMode: false
  maxAutoStagedPerTick: 10  # cap per scheduler tick

And bump the TTL if a weekend might pass between stage + approve:

approvals:
  ttlSeconds: 172800  # 48h

Approved a recommendation but nothing happened in the cluster

Check the Approvals view for a "Approved, but the write didn't land" banner — the dispatch failed post-approval. The audit log has the specific error:

kubectl get cm -n tiphys tiphys-audit -o jsonpath='{.data.audit\.json}' | jq '.[-5:]'

Common causes:

• Core Plus entitlement lapsed between stage + approve. Re-check the Modules view.
• RBAC changed. Re-look at the writer ClusterRole diff.
• The target workload was deleted between stage + approve (the dry-run from 12 hours ago was against a now-gone resource).

UI

Where's the admin password?

Fresh installs create admin with a one-time password on first start:

kubectl -n tiphys exec tiphys-0 -c backend -- cat /var/lib/clu/initial-admin-password

No such file means it has already been used and replaced. Reset it:

kubectl -n tiphys exec tiphys-0 -c backend -- python -m ops_agent.reset_admin

That prints a new one-time password and signs everyone out. If the password comes from auth.builtin.existingSecret, change it in the Secret and restart the pod instead.

"Too many attempts. Try again in N seconds."

Five wrong passwords in a row lock admin for 30 seconds, doubling with each further miss up to 15 minutes. Wait it out, or restart the pod to clear the counter.

The sign-in page says "Tiphys couldn't set up sign-in"

The backend had nowhere writable for the one-time password (the state volume and /tmp both failed). Check the backend log and the state PVC, then restart the pod. Alternatively set auth.builtin.existingSecret.

The UI says "Sign-in happens at your proxy"

The install runs with auth.mode=proxy and the request carried no X-Forwarded-User header, as with a plain port-forward. Open Tiphys through your SSO ingress, or switch to the built-in sign-in with --set auth.mode=builtin.

UI shows "Failed to load approvals" / "Failed to load reports"

The ConfigMap client can't read the store. Verify the state Role exists + is bound:

kubectl get role -n tiphys | grep state
kubectl get rolebinding -n tiphys | grep state

Missing → re-render the chart (the role-state.yaml template is part of the default install). Present but the API server still 403s → look for custom NetworkPolicy or AdmissionController rules blocking the agent's SA.

SSE chat stream disconnects after 60 seconds

Ingress timeout. Configure long-lived timeouts for /api/query:

• nginx: proxy-read-timeout: 3600 on the Ingress annotation.
• ALB: idle_timeout.timeout_seconds=3600 on the load balancer attributes.

See UI Access for the full examples.

Performance

First chat response is slow

First conversation turn builds the system prompt, which reads the scan. If the scanner hasn't run yet, the prompt has no cluster context and the agent will call more tools to compensate. Wait for the first scheduled knowledge refresh (or force a scan via run a cluster scan), then chat is snappier.

Reporter scan blocks for minutes

Large clusters (10k+ pods) can spend significant time listing resources. Reduce the scanner's surface via Helm:

agent:
  namespaces:
    include: [prod, staging]      # Only these namespaces

or

agent:
  namespaces:
    exclude: [kube-system, monitoring, istio-system, argo, argocd]

Bedrock latency feels high

Switch the interactive model tier to fast (Haiku) if the workload is tool-call-heavy rather than reasoning-heavy:

agent:
  model:
    interactive: fast

Haiku is ~3-5x faster than Sonnet; the tradeoff is weaker multi-step reasoning. Fine for "fetch logs + describe pod" flows; worse for "debug this cross-cluster connectivity puzzle."

Logs + debugging

# Backend logs (streaming)
kubectl logs -n tiphys deploy/tiphys -c backend -f

# Frontend / ingress logs
kubectl logs -n tiphys deploy/tiphys -c frontend

# State ConfigMaps
kubectl get cm -n tiphys -o name | grep tiphys-
# → tiphys-reports
# → tiphys-approvals
# → tiphys-snoozes
# → tiphys-audit
# → tiphys-scan

# Inspect the audit log
kubectl get cm -n tiphys tiphys-audit \
  -o jsonpath='{.data.audit\.json}' | jq '.[-20:]'

Still stuck

• Check Getting Started for the happy-path walkthrough.
• Re-read Install Tiphys if anything AWS-shaped is failing.
• File an issue with the commit SHA (kubectl get deploy -n tiphys tiphys -o jsonpath='{.metadata.annotations}'), the pod logs, and the relevant kubectl describe output.