Skip to content

Monitoring and troubleshooting

What the deployment tells you about itself, and the failures already seen. Health probes, Kubernetes conditions, the console, the audit log and JSON logs exist. No metrics exporter, Prometheus endpoint or ServiceMonitor exists anywhere in the repository.

1. Health endpoints

Component Endpoint Answer Probe in chart
plane GET /healthz always 200 while up; touches nothing none
plane GET /.well-known/troupe discovery: issuer, client id, endpoints, scopes, plane name, the web app's address, protocol version and build (commit, time, version) startup 2 s × 30, readiness 5 s, liveness 10 s
plane GET /.well-known/jwks.json the signing keys, or 503 when OpenBao or its credential is unreachable — the quickest OpenBao check none
worker GET /health/live, GET /health/ready ok; ready, or 503 draining so its Service stops routing liveness and readiness every 5 s
a2a GET /healthz 200 startup, readiness, liveness
gui GET /healthz 200 readiness, liveness
operator exec /app/bin/troupe_operator pid non-zero when no node answers liveness every 30 s

If the build in /.well-known/troupe (also in the console's footer) is not the commit you deployed, the rollout has not finished or the image tag did not move.

2. Kubernetes signals

  • WorkerProfile conditions Ready, PolicyViolation, SecretMissing, MCPIdentityMissing, UpgradePending, EgressByHostname, PlaneCAMissing, EndpointUnreachable, IngressNamespaceMissing (profiles-and-policy.md §2); kubectl -n troupe-system get wp.
  • Events: the operator may write them (through Bonny); Troupe's own code writes none.
  • PodDisruptionBudgets: the plane's minAvailable: 1 with more than one replica, each profile's maxUnavailable: 1. A node drain that hangs is one of these doing its job.
  • The operator's leader Lease in troupe-system.

3. The console

The console at /console (console.md) opens on the overview: what needs attention, per profile its pods and load, per team its spend against its budget, and session counts, read again every ten seconds. Pods, sessions, bundles (with adoption per pod), triggers, audit and settings each have a screen. Every screen is a call into Troupe.Plane.Admin, and admin.overview returns the same map to /rpc and /mcp.

4. Audit log

Every administrative change is an audit_events row: actor, action, subject_kind, subject_id, a detail diff keyed by dotted path (spec.llm.model) and the time. Values whose key looks like a secret are redacted. Break-glass writes admin.breakglass on success and on refusal. A refused change writes nothing. admin.audit.list filters by actor, kind, subject_id and limit. A team admin reads the rows about the teams they administer (the team, its triggers, principals and sessions) and none about the plane as a whole; a platform admin reads them all. admin.audit.verify checks the trail's integrity and answers the first row that does not verify, with its reason.

5. Logs

TROUPE_LOG_FORMAT=json on the plane gives one JSON object per line with request_id and session_id. There is no Helm value: add the variable to the Deployment yourself. The operator, workers and A2A log in Elixir's default text format. Lines worth alerting on:

Line (prefix troupe plane: unless noted) Meaning
no OpenBao credential (...) the plane can neither sign tokens nor publish its JWKS
troupe worker: no OpenBao token (...) a worker pod could not log in under the role it names; a role per profile that was never made reads refused with HTTP 400
break-glass token accepted from <ip> / break-glass token refused from <ip> the door (POST /auth/breakglass) was used or probed
no TroupePolicy is configured or reachable; every MCP host is allowed the plane has no Kubernetes connection; bundle egress checks are open
<ns>/<pod> stopped heartbeating; ... a pod missed its 15 s lease
ledger drift of N micros — ... the reconcile found disagreement
scheduler could not fire <trigger> ... a cron firing failed to create its session
erasure of <id> waits for a pod: ... no healthy pod could take a team session's copy and objects; it runs at the next enrolment, the key already gone
erasure of <id> is pending: its key was not destroyed: ... OpenBao refused the plane, or could not be reached; the session reads erasure_pending until an erase is retried, the plane's next erasure pass, or for a private session its owner's daemon next connecting
erasure pass destroyed N session key(s)[; M could not be destroyed ...] the five-minute pass finished erasures whose key was refused before, or, after an upgrade to 0.9.0, destroyed the keys earlier erasures of team sessions left; M is tried again every five minutes, and is said again when it changes
troupe operator: refusing <profile>: <violations> a policy violation; nothing was created
troupe operator: pruning <kind>/<name> a scale-down, a revoked grant or ciliumAvailable switched off took an object away
sandbox: this worker runs every command in a sandbox and …, so no command will run the worker cannot start bubblewrap, and refuses every command until it can (Decision 832)

troupe_core also emits two :telemetry events, [:troupe, :llm, :retry], which troupe bench --live counts, and [:troupe, :llm, :unpriced], and the plane's endpoint Phoenix's own; nothing in a deployment attaches to them.

6. Fleet mechanics

Mechanism Numbers
Worker heartbeat every 5 s over the control channel: capacity, bundle hash, version, active sessions, disk, draining
Lease no heartbeat for 15 s and a pod is unhealthy and not placed on
Placement healthy, not draining, disk below 80 %
Scale-down after 2 min wanting fewer pods: the pods above the new count drained, turns given up to 5 min, the count lowered once they hold no active session
Pod stop a worker stopped without a drain drains itself: turns get 150 s, then every session is put to sleep inside the grace period
Worker disk caches evicted above 70 %, checked every 30 s
Dormancy a session sleeps after 10 min idle on the pod; sealed every 60 s and at a turn's end
Caches ledger sums 60 s per node; settings 5 s; provider JWKS until a bad signature

A drain (a scale-down, an upgrade, or admin.pod.drain) stops placing on the pod, lets running turns finish for up to the drain timeout (300 s by default), cancels any still running, then puts every session to sleep one after another, each given up to 120 s to seal and upload. The pod tells the plane before each session how many it has left, and the plane waits as long as it hears that, up to a ceiling of the timeout, 120 s a session and 30 s more: about 11.5 min for three busy sessions. A pod that goes quiet for 150 s (and at least the timeout and 30 s) or reaches the ceiling is given up on: its sessions are marked dormant at their last seal, and the audit has a pod.drain.gave_up row saying which, why, the ceiling and how long the plane waited. A pod the scale-down removed is listed gone once it is quiet past its lease, and the sweep forgets its row once it is quiet past the drain timeout and holds no active session. Once the plane records a pod behind on an upgrade drained, the operator replaces it within seconds, if the pod has stopped being Ready.

7. If something does not work

Symptom Cause
Pod OOMKilled in a second, empty log the BEAM sized its port table from RLIMIT_NOFILE; set +Q in any pod spec you write yourself
A release dies in its config provider Kubernetes injected <SERVICE>_PORT=tcp://…, which collides with TROUPE_PLANE_CONTROL_PORT; keep enableServiceLinks: false
A worker enrols, looks healthy and refuses every client it has no JWKS; the plane pushes one at enrolment and logs if that failed
permission denied logging into OpenBao no reviewer JWT on the Kubernetes auth mount
A worker Ingress answers 503 the ingress namespace lacks troupe.dev/ingress=true: the profile's IngressNamespaceMissing condition, and the Overview's Needs attention, give the kubectl label namespace line (configuration.md Part E); or the pod is drained and was never deleted
the pod did not accept the session, a signer crash on nil object-store credentials missing from the worker namespace: SecretMissing says why the operator could not copy them
the pod did not accept the session with object_store_unreachable; in a worker's log, could not read with it, or this pod cannot use its object store when it enrols the pod cannot reach objectStore.endpoint: check its egress. With Cilium, the troupe-egress CiliumNetworkPolicy in troupe-w-<profile> names the host when it is outside the cluster
the pod did not accept the session with kms_unreachable, or could not read with it in a worker's log the pod cannot reach bao.address: check its egress, as for the object store above
Workers enrol, then go quiet the worker namespace lacks troupe.dev/workers=true
429 from the plane behind one office address plane.ingress.rateLimit is per client IP; raise connections first
unauthorized pulling an image in a worker namespace the pull secret is not named in imagePullSecrets, so the operator does not copy it
Worker pod CreateContainerConfigError the llm.secretRef Secret is missing in troupe-w-<profile>: SecretMissing says where it was looked for, and for one in troupe-system without troupe.dev/worker-secret=true gives the kubectl label line
PlaneCAMissing: True the operator reads no certificate from TROUPE_PLANE_CONTROL_CA yet: the plane.controlTls Secret has no ca.crt (cert-manager has not issued it, or a Secret of your own lacks it); the operator's log says the same once a pass
A profile is saved but no WorkerProfile appears; not_applied, no_cluster the plane has no Kubernetes connection (TROUPE_KUBECONFIG, or the in-pod ServiceAccount)
PolicyViolation: NoPolicy on every profile policy.name is not default, and nothing sets TROUPE_POLICY_NAME
A profile keeps a pod more than its sessions need a scale-down lowers the count only once the pods above it hold no active session. did not drain for a scale-down: {:stranded, [...]} in the plane's log names a session the pod never reported dormant; the pod stays until it is
Pods stay on the old image, UpgradePending: True a pod is replaced once it holds no active session, one at a time; the message says which pod waits and for what. waits to be drained for long: its sessions are still in use, so admin.pod.drain it to roll it now. A pod the plane recorded in troupe.dev/drained that is still Ready: drain it again (profiles-and-policy.md §3)
The console signs you in but shows only Mine the admin group is not among the groups the token carries, or groups_claim names the wrong claim; Sign-in's identity check says how many people carry it
The exchange says the token has no id_token openid not in scope, or the registration cannot issue id tokens
Every sign-in fails before a password, AADSTS650053 groups asked for as a scope; take it out of plane.oidc.scopes
/.well-known/jwks.json 503; troupe login fails at exchange no OpenBao credential: no static token and the projected token unreadable or refused
/mcp 401 with a valid provider token its aud is none of the client id, api://<client id>, <base_url>/mcp, or the issuer differs (v1 against v2 endpoints)
MCP OAuth fails after consent, AADSTS9010010 the advertised scope and the client's resource name different resources; expose <base_url>/mcp/admin or set plane.oidc.mcpScope
The provider refuses the console's or the GUI's redirect URI <base_url>/console or <base_url>/app is not registered as a single-page application's (integrations.md)
troupe.ledger.reconcile says :no_gateway_configured :troupe_plane, :gateway is set by nothing here
Workers cannot write to a bucket outside us-east-1, the plane can the worker pods have no TROUPE_OBJECT_REGION and sign for us-east-1: an operator from before it passed objectStore.region on to them
Plane replicas do not cluster during a rolling image upgrade the Erlang cookie is baked into each image build
Team volumes never appear inside sessions they are mounted on the pod, not into a session's mount table
Every command on a profile's pods fails, "this worker runs every command in a sandbox" the worker container's seccomp profile refuses bubblewrap its user namespace (worker.seccomp.type: RuntimeDefault), the nodes do not allow unprivileged user namespaces, or the image lacks bubblewrap (worker-sandbox.md)
Every command on a profile's pods fails, "bubblewrap could not start one here (bwrap: ...)" what bubblewrap names refused it: the table at the end of the workers' sandbox says what each message means and what fixes it (Can't mount proc: the pod has no user namespace of its own; Failed to make / slave or setting up uid map: AppArmor, worker.appArmor.enabled: false)
Worker pods are not created, FailedCreate on the StatefulSet naming procMount or seccompProfile the worker namespace's Pod Security level is baseline or restricted; workers need privileged, which the operator labels it with unless podSecurity.labels is off or something else rewrote the label (worker-sandbox.md)
The plane's log or the console's Overview says its OpenBao role "may not" do something the role lacks that grant; the line comes with the bao lines an OpenBao administrator runs to add it (openbao.md)
Worker pods stay in ContainerCreating or CreateContainerError naming an idmapped mount a volume's filesystem does not take idmapped mounts, as NFS does not (worker-sandbox.md)
Worker pods stay in CreateContainerError, "cannot load seccomp profile" the Localhost profile is not on that node: kubectl -n troupe-system get daemonset troupe-worker-profiles, or where something else installs it, its path against worker.seccomp.localhostProfile
Worker pods stay in CreateContainerError, "apparmor profile not found" the AppArmor profile is not loaded on that node: kubectl -n troupe-system get daemonset troupe-worker-profiles and the log of its apparmor container, or where something else loads it, its name against worker.appArmor.profile
Worker pods fail at once, "Cannot enforce AppArmor: AppArmor is not enabled on the host" the node runs no AppArmor and the worker asks for a profile: worker.appArmor.enabled: false (worker-sandbox.md)
A client is refused with update_required it is below the minimum the console's Client versions sets, or on a version blocked there (routine-tasks.md)