TIPHYSby Intelligant AI

Troubleshooting

Common install + runtime issues and their fixes. Tiphys's chat is the fastest troubleshooting surface for most of these — the agent already knows how to diagnose its own errors — but the common symptoms are collected here as a pre-chat reference.

Install

Pod stuck in Pending after helm install

  • Resource requests don't fit the node: check kubectl describe pod -n tiphys — the scheduler's FailedScheduling event names the exact constraint that failed.

  • No default StorageClass: the state volume never binds. Since v0.5.0 helm install stops with "This cluster has no default StorageClass" instead of leaving the pod in Pending. Pick a class from kubectl get storageclass and set persistence.storageClass, mark one as the default, or install a provisioner such as local-path-provisioner. (persistence.mode=configmap needs no volume, but then the admin password has to come from auth.builtin.existingSecret.)

A wrong IRSA annotation doesn't cause Pending; see the next section.

Pod is Running but AWS calls are denied

The ServiceAccount annotation doesn't match a real IAM role, or the role's trust policy doesn't trust this ServiceAccount. The pod still starts (Tiphys's permission checks never stop it), but every AWS call fails. With Bedrock as the model provider, chat replies fail with an AccessDenied error from AssumeRoleWithWebIdentity, and the backend log shows llm stream failed. Fix the annotation and restart the pod:

kubectl annotate sa -n tiphys tiphys \
  eks.amazonaws.com/role-arn=arn:aws:iam::...:role/tiphys-irsa --overwrite
kubectl rollout restart -n tiphys sts/tiphys

With persistence.mode=configmap the workload is a Deployment: use deploy/tiphys here and in the commands below.

Pod in CrashLoopBackOff

kubectl logs -n tiphys sts/tiphys -c backend
  • pydantic_core._pydantic_core.ValidationError on startup — a Helm value doesn't match the expected type. The traceback names the field.
  • Missing AWS credentials don't crash the pod: see Pod is Running but AWS calls are denied.

Chat returns errors

error_kind: permission_denied in chat

The agent hit a 403 — K8s RBAC, Prometheus auth, or AWS IAM. Tiphys surfaces the exact fix inline. Example:

error_kind: permission_denied
The agent was denied 'list' on PodTemplate (namespace 'prod'): forbidden.

Fix:
Add this rule to the reader ClusterRole (...):
- apiGroups: [""]
  resources:
    - podtemplates
  verbs: ['get', 'list', 'watch']

Secrets are the exception: don't paste a Secret rule into the reader. Secret read is off on purpose and turned on only with rbac.helmValuesDecode=true; read Data residency by inference provider first.

The hint can end with a rollout step that names scripts/dev-deploy.sh. That script is in the Tiphys source repository, not in your install: on your cluster, roll the rule out with a helm upgrade from your copy of the chart.

The Fix: block is paste-ready — apply it to the policy named in the hint. Same shape for:

  • K8s RBAC → paste into your customer-side fork of the reader ClusterRole (the rules are inlined in Permissions).
  • Prometheus 401/403 → set integrations.prometheus.url to the unauthenticated in-cluster Service, or open a NetworkPolicy.
  • AWS IAM → paste into the Cloud IAM policy (the JSON is inlined in IAM setup).

tool execution error: Prometheus is not configured...

You're running Tiphys without Prometheus configured (or auto-discovery didn't find it). The error names both fixes:

  1. Install a kube-prometheus-stack release — Tiphys auto-detects Services whose name contains prometheus with port 9090.

  2. Set the URL explicitly — if your Prometheus runs under a non-standard name:

    integrations:
  prometheus:
    url: http://my-prometheus.monitoring.svc:9090

Bedrock returns AccessDeniedException

Two distinct failure modes:

  • IAM action missing — the IRSA role doesn't have bedrock:InvokeModel. Fix via Terraform or IAM console.
  • Model access not requested — Bedrock in a fresh AWS account blocks every model by default until you opt in. Go to the Bedrock console → Model access → enable Claude Sonnet + Haiku in the target region.

The error message distinguishes them: "is not authorized" → IAM action missing; "You don't have access to the model" → model access not requested.

Reports + approvals

Reports view is empty

First check takes up to 30 minutes if the pod was restarted immediately after install (the first scheduled tick hasn't fired yet). Force one from the UI (Run now) or from chat ("run a health check now"). The endpoint behind both is POST /api/reports/run-now.

Approvals view is empty overnight

By default reports.recommenderMode=true — findings carry recommendations but the operator has to manually stage them. For "approvals populate overnight" behavior, set:

reports:
  recommenderMode: false
  maxAutoStagedPerTick: 10  # cap per scheduler tick

And bump the TTL if a weekend might pass between stage + approve:

approvals:
  ttlSeconds: 172800  # 48h

Approved a recommendation but nothing happened in the cluster

Check the Approvals view for a "Approved, but the write didn't land" banner — the dispatch failed post-approval. The audit log (the Audit view) has the specific error. To read the latest entries from a shell, with the default SQLite store:

kubectl -n tiphys exec tiphys-0 -c backend -- python -c \
  "import sqlite3; db = sqlite3.connect('file:/var/lib/tiphys/state.db?mode=ro', uri=True); [print(d) for (d,) in db.execute('SELECT data FROM audit ORDER BY timestamp DESC LIMIT 5')]"

With persistence.mode=configmap:

kubectl get cm -n tiphys tiphys-audit -o jsonpath='{.data.audit\.json}' | jq '.[-5:]'

Common causes:

  • Core Plus entitlement lapsed between stage + approve. Re-check the Modules view.
  • RBAC changed. Re-look at the writer ClusterRole diff.
  • The target workload was deleted between stage + approve (the dry-run from 12 hours ago was against a now-gone resource).

UI

Where's the admin password?

Fresh installs create admin with a one-time password on first start:

kubectl -n tiphys exec tiphys-0 -c backend -- cat /var/lib/tiphys/initial-admin-password

No such file means it has already been used and replaced. Reset it:

kubectl -n tiphys exec tiphys-0 -c backend -- python -m ops_agent.reset_admin

That prints a new one-time password and signs everyone out. If the password comes from auth.builtin.existingSecret, change it in the Secret and restart the pod instead.

"Too many attempts. Try again in N seconds."

Five wrong passwords in a row lock admin for 30 seconds, doubling with each further miss up to 15 minutes. Wait it out, or restart the pod to clear the counter.

The sign-in page says "Tiphys couldn't set up sign-in"

The backend had nowhere writable for the one-time password (the state volume and /tmp both failed). Check the backend log and the state PVC, then restart the pod. Alternatively set auth.builtin.existingSecret.

The UI says "Sign-in happens at your proxy"

The install runs with auth.mode=proxy and the request carried no X-Forwarded-User header, as with a plain port-forward. Open Tiphys through your SSO ingress, or switch to the built-in sign-in with --set auth.mode=builtin.

UI shows "Failed to load approvals" / "Failed to load reports"

The backend couldn't read the store. Where that is depends on persistence.mode.

With the default sqlite, the stores are in /var/lib/tiphys/state.db on the state PVC. Check the backend log for the SQLite error, and check that the PVC is Bound and not full:

kubectl get pvc -n tiphys
kubectl -n tiphys exec tiphys-0 -c backend -- df -h /var/lib/tiphys

With configmap, the ConfigMap client can't read the store. Verify the state Role exists + is bound:

kubectl get role -n tiphys | grep state
kubectl get rolebinding -n tiphys | grep state

Missing → re-render the chart (the role-state.yaml template is part of the default install). Present but the API server still 403s → look for custom NetworkPolicy or AdmissionController rules blocking the agent's SA.

SSE chat stream disconnects after 60 seconds

Ingress timeout. Configure long-lived timeouts for /api/query:

  • nginx: proxy-read-timeout: 3600 on the Ingress annotation.
  • ALB: idle_timeout.timeout_seconds=3600 on the load balancer attributes.

See UI Access for the full examples.

Performance

First chat response is slow

First conversation turn builds the system prompt, which reads the scan. If the scanner hasn't run yet, the prompt has no cluster context and the agent will call more tools to compensate. Wait for the first scheduled knowledge refresh (or force a scan via run a cluster scan), then chat is snappier.

Reporter scan blocks for minutes

Large clusters (10k+ pods) can spend significant time listing resources, and a scan reads every namespace Tiphys's reader can. Scan less often:

agent:
  schedule:
    knowledgeGraphRefresh: 6h     # default 1h
    healthCheckInterval: 2h       # default 30m

Chat feels slow

Chat uses the fast tier by default. If you set the interactive tier to smart, set it back for tool-call-heavy work:

llm:
  tiers:
    interactive: fast

The fast model (Claude Haiku, by default) is several times faster than the smart one (Claude Sonnet); the tradeoff is weaker multi-step reasoning. Fine for "fetch logs + describe pod" flows; worse for "debug this cross-cluster connectivity puzzle."

Logs + debugging

The backend logs at info. For more while you chase a problem, turn on debug, reproduce it, then turn it back off:

helm upgrade tiphys oci://ghcr.io/intelligant-ai/charts/tiphys -n tiphys \
  --reuse-values --set agent.logLevel=debug
kubectl logs -n tiphys sts/tiphys -c backend -f | grep DEBUG
helm upgrade tiphys oci://ghcr.io/intelligant-ai/charts/tiphys -n tiphys \
  --reuse-values --set agent.logLevel=info

Debug adds one line per tool call (tool k8s_call (verb=get kind=Pod namespace=prod): 2048 bytes in 41 ms), per AWS call (aws ec2 DescribeVolumes (us-east-1): 403 AccessDenied in 120 ms) and per helm run (its command line, exit code and time), plus the reads Tiphys tried and couldn't make. It never logs manifests, values, prompts or secrets, and libraries stay at info. Pin --version to the chart you run, or the upgrade moves you to the latest.

# Backend logs (streaming)
kubectl logs -n tiphys sts/tiphys -c backend -f

# Frontend / ingress logs
kubectl logs -n tiphys sts/tiphys -c frontend

# State ConfigMaps
kubectl get cm -n tiphys -o name | grep tiphys-
# → tiphys-config         (chart settings, not state)
# → tiphys-scan
# → tiphys-prices         (once AWS list prices have been looked up)
# → tiphys-billed-costs   (once Cost Explorer has been called)
# With persistence.mode=configmap, also:
# → tiphys-reports
# → tiphys-approvals
# → tiphys-snoozes
# → tiphys-audit
# → tiphys-conversations

# Inspect the audit log (persistence.mode=configmap; with the default
# SQLite store, use the command under "Approved a recommendation")
kubectl get cm -n tiphys tiphys-audit \
  -o jsonpath='{.data.audit\.json}' | jq '.[-20:]'

Still stuck

  • Check Getting Started for the happy-path walkthrough.
  • Re-read Install Tiphys if anything AWS-shaped is failing.
  • File an issue with the Tiphys version (kubectl get sts -n tiphys tiphys -o jsonpath='{.metadata.labels.app\.kubernetes\.io/version}'), the pod logs, and the relevant kubectl describe output.
PreviousValidation