Monitoring and troubleshooting¶
What the deployment tells you about itself, and the failures already seen. Health probes, Kubernetes conditions, the console, the audit log and JSON logs exist. No metrics exporter, Prometheus endpoint or ServiceMonitor exists anywhere in the repository.
1. Health endpoints¶
| Component | Endpoint | Answer | Probe in chart |
|---|---|---|---|
| plane | GET /healthz |
always 200 while up; touches nothing | none |
| plane | GET /.well-known/troupe |
discovery: issuer, client id, endpoints, scopes, plane name, the web app's address, protocol version and build (commit, time, version) | startup 2 s × 30, readiness 5 s, liveness 10 s |
| plane | GET /.well-known/jwks.json |
the signing keys, or 503 when OpenBao or its credential is unreachable — the quickest OpenBao check | none |
| worker | GET /health/live, GET /health/ready |
ok; ready, or 503 draining so its Service stops routing |
liveness and readiness every 5 s |
| a2a | GET /healthz |
200 | startup, readiness, liveness |
| gui | GET /healthz |
200 | readiness, liveness |
| operator | exec /app/bin/troupe_operator pid |
non-zero when no node answers | liveness every 30 s |
If the build in /.well-known/troupe (also in the console's footer) is not the commit you
deployed, the rollout has not finished or the image tag did not move.
2. Kubernetes signals¶
WorkerProfileconditionsReady,PolicyViolation,SecretMissing,MCPIdentityMissing,UpgradePending,EgressByHostname,PlaneCAMissing,EndpointUnreachable,IngressNamespaceMissing(profiles-and-policy.md §2);kubectl -n troupe-system get wp.- Events: the operator may write them (through Bonny); Troupe's own code writes none.
- PodDisruptionBudgets: the plane's
minAvailable: 1with more than one replica, each profile'smaxUnavailable: 1. A node drain that hangs is one of these doing its job. - The operator's leader Lease in
troupe-system.
3. The console¶
The console at /console (console.md) opens on the overview: what needs attention,
per profile its pods and load, per team its spend against its budget, and session counts,
read again every ten seconds. Pods, sessions, bundles (with adoption per pod),
triggers, audit and settings each have a screen. Every screen is a call into
Troupe.Plane.Admin, and admin.overview returns the same map to /rpc and /mcp.
4. Audit log¶
Every administrative change is an audit_events row: actor, action, subject_kind,
subject_id, a detail diff keyed by dotted path (spec.llm.model) and the time. Values
whose key looks like a secret are redacted. Break-glass writes admin.breakglass on success
and on refusal. A refused change writes nothing. admin.audit.list filters by actor,
kind, subject_id and limit. A team admin reads the rows about the teams they
administer (the team, its triggers, principals and sessions) and none about the plane as a
whole; a platform admin reads them all. admin.audit.verify checks the trail's integrity
and answers the first row that does not verify, with its reason.
5. Logs¶
TROUPE_LOG_FORMAT=json on the plane gives one JSON object per line with request_id
and session_id. There is no Helm value: add the variable to the Deployment yourself. The
operator, workers and A2A log in Elixir's default text format. Lines worth alerting on:
Line (prefix troupe plane: unless noted) |
Meaning |
|---|---|
no OpenBao credential (...) |
the plane can neither sign tokens nor publish its JWKS |
troupe worker: no OpenBao token (...) |
a worker pod could not log in under the role it names; a role per profile that was never made reads refused with HTTP 400 |
break-glass token accepted from <ip> / break-glass token refused from <ip> |
the door (POST /auth/breakglass) was used or probed |
no TroupePolicy is configured or reachable; every MCP host is allowed |
the plane has no Kubernetes connection; bundle egress checks are open |
<ns>/<pod> stopped heartbeating; ... |
a pod missed its 15 s lease |
ledger drift of N micros — ... |
the reconcile found disagreement |
scheduler could not fire <trigger> ... |
a cron firing failed to create its session |
erasure of <id> waits for a pod: ... |
no healthy pod could take a team session's copy and objects; it runs at the next enrolment, the key already gone |
erasure of <id> is pending: its key was not destroyed: ... |
OpenBao refused the plane, or could not be reached; the session reads erasure_pending until an erase is retried, the plane's next erasure pass, or for a private session its owner's daemon next connecting |
erasure pass destroyed N session key(s)[; M could not be destroyed ...] |
the five-minute pass finished erasures whose key was refused before, or, after an upgrade to 0.9.0, destroyed the keys earlier erasures of team sessions left; M is tried again every five minutes, and is said again when it changes |
troupe operator: refusing <profile>: <violations> |
a policy violation; nothing was created |
troupe operator: pruning <kind>/<name> |
a scale-down, a revoked grant or ciliumAvailable switched off took an object away |
sandbox: this worker runs every command in a sandbox and …, so no command will run |
the worker cannot start bubblewrap, and refuses every command until it can (Decision 832) |
troupe_core also emits two :telemetry events, [:troupe, :llm, :retry], which
troupe bench --live counts, and [:troupe, :llm, :unpriced], and the plane's endpoint
Phoenix's own; nothing in a deployment attaches to them.
6. Fleet mechanics¶
| Mechanism | Numbers |
|---|---|
| Worker heartbeat | every 5 s over the control channel: capacity, bundle hash, version, active sessions, disk, draining |
| Lease | no heartbeat for 15 s and a pod is unhealthy and not placed on |
| Placement | healthy, not draining, disk below 80 % |
| Scale-down | after 2 min wanting fewer pods: the pods above the new count drained, turns given up to 5 min, the count lowered once they hold no active session |
| Pod stop | a worker stopped without a drain drains itself: turns get 150 s, then every session is put to sleep inside the grace period |
| Worker disk | caches evicted above 70 %, checked every 30 s |
| Dormancy | a session sleeps after 10 min idle on the pod; sealed every 60 s and at a turn's end |
| Caches | ledger sums 60 s per node; settings 5 s; provider JWKS until a bad signature |
A drain (a scale-down, an upgrade, or admin.pod.drain) stops placing on the pod, lets
running turns finish for up to the drain timeout (300 s by default), cancels any still running, then
puts every session to sleep one after another, each given up to 120 s to seal and upload.
The pod tells the plane before each session how many it has left, and the plane waits as
long as it hears that, up to a ceiling of the timeout, 120 s a session and 30 s more: about
11.5 min for three busy sessions. A pod that goes quiet for 150 s (and at least the timeout
and 30 s) or reaches the ceiling is given up on: its sessions are marked dormant at their
last seal, and the audit has a pod.drain.gave_up row saying which, why, the ceiling and
how long the plane waited. A pod the scale-down removed is listed gone once it is quiet past its lease, and the sweep forgets its row once it is quiet past the drain timeout and holds no active session. Once the plane records a pod behind on an upgrade drained, the
operator replaces it within seconds, if the pod has stopped being Ready.
7. If something does not work¶
| Symptom | Cause |
|---|---|
| Pod OOMKilled in a second, empty log | the BEAM sized its port table from RLIMIT_NOFILE; set +Q in any pod spec you write yourself |
| A release dies in its config provider | Kubernetes injected <SERVICE>_PORT=tcp://…, which collides with TROUPE_PLANE_CONTROL_PORT; keep enableServiceLinks: false |
| A worker enrols, looks healthy and refuses every client | it has no JWKS; the plane pushes one at enrolment and logs if that failed |
permission denied logging into OpenBao |
no reviewer JWT on the Kubernetes auth mount |
| A worker Ingress answers 503 | the ingress namespace lacks troupe.dev/ingress=true: the profile's IngressNamespaceMissing condition, and the Overview's Needs attention, give the kubectl label namespace line (configuration.md Part E); or the pod is drained and was never deleted |
the pod did not accept the session, a signer crash on nil |
object-store credentials missing from the worker namespace: SecretMissing says why the operator could not copy them |
the pod did not accept the session with object_store_unreachable; in a worker's log, could not read with it, or this pod cannot use its object store when it enrols |
the pod cannot reach objectStore.endpoint: check its egress. With Cilium, the troupe-egress CiliumNetworkPolicy in troupe-w-<profile> names the host when it is outside the cluster |
the pod did not accept the session with kms_unreachable, or could not read with it in a worker's log |
the pod cannot reach bao.address: check its egress, as for the object store above |
| Workers enrol, then go quiet | the worker namespace lacks troupe.dev/workers=true |
429 from the plane behind one office address |
plane.ingress.rateLimit is per client IP; raise connections first |
unauthorized pulling an image in a worker namespace |
the pull secret is not named in imagePullSecrets, so the operator does not copy it |
Worker pod CreateContainerConfigError |
the llm.secretRef Secret is missing in troupe-w-<profile>: SecretMissing says where it was looked for, and for one in troupe-system without troupe.dev/worker-secret=true gives the kubectl label line |
PlaneCAMissing: True |
the operator reads no certificate from TROUPE_PLANE_CONTROL_CA yet: the plane.controlTls Secret has no ca.crt (cert-manager has not issued it, or a Secret of your own lacks it); the operator's log says the same once a pass |
A profile is saved but no WorkerProfile appears; not_applied, no_cluster |
the plane has no Kubernetes connection (TROUPE_KUBECONFIG, or the in-pod ServiceAccount) |
PolicyViolation: NoPolicy on every profile |
policy.name is not default, and nothing sets TROUPE_POLICY_NAME |
| A profile keeps a pod more than its sessions need | a scale-down lowers the count only once the pods above it hold no active session. did not drain for a scale-down: {:stranded, [...]} in the plane's log names a session the pod never reported dormant; the pod stays until it is |
Pods stay on the old image, UpgradePending: True |
a pod is replaced once it holds no active session, one at a time; the message says which pod waits and for what. waits to be drained for long: its sessions are still in use, so admin.pod.drain it to roll it now. A pod the plane recorded in troupe.dev/drained that is still Ready: drain it again (profiles-and-policy.md §3) |
| The console signs you in but shows only Mine | the admin group is not among the groups the token carries, or groups_claim names the wrong claim; Sign-in's identity check says how many people carry it |
The exchange says the token has no id_token |
openid not in scope, or the registration cannot issue id tokens |
Every sign-in fails before a password, AADSTS650053 |
groups asked for as a scope; take it out of plane.oidc.scopes |
/.well-known/jwks.json 503; troupe login fails at exchange |
no OpenBao credential: no static token and the projected token unreadable or refused |
/mcp 401 with a valid provider token |
its aud is none of the client id, api://<client id>, <base_url>/mcp, or the issuer differs (v1 against v2 endpoints) |
MCP OAuth fails after consent, AADSTS9010010 |
the advertised scope and the client's resource name different resources; expose <base_url>/mcp/admin or set plane.oidc.mcpScope |
| The provider refuses the console's or the GUI's redirect URI | <base_url>/console or <base_url>/app is not registered as a single-page application's (integrations.md) |
troupe.ledger.reconcile says :no_gateway_configured |
:troupe_plane, :gateway is set by nothing here |
Workers cannot write to a bucket outside us-east-1, the plane can |
the worker pods have no TROUPE_OBJECT_REGION and sign for us-east-1: an operator from before it passed objectStore.region on to them |
| Plane replicas do not cluster during a rolling image upgrade | the Erlang cookie is baked into each image build |
| Team volumes never appear inside sessions | they are mounted on the pod, not into a session's mount table |
| Every command on a profile's pods fails, "this worker runs every command in a sandbox" | the worker container's seccomp profile refuses bubblewrap its user namespace (worker.seccomp.type: RuntimeDefault), the nodes do not allow unprivileged user namespaces, or the image lacks bubblewrap (worker-sandbox.md) |
| Every command on a profile's pods fails, "bubblewrap could not start one here (bwrap: ...)" | what bubblewrap names refused it: the table at the end of the workers' sandbox says what each message means and what fixes it (Can't mount proc: the pod has no user namespace of its own; Failed to make / slave or setting up uid map: AppArmor, worker.appArmor.enabled: false) |
Worker pods are not created, FailedCreate on the StatefulSet naming procMount or seccompProfile |
the worker namespace's Pod Security level is baseline or restricted; workers need privileged, which the operator labels it with unless podSecurity.labels is off or something else rewrote the label (worker-sandbox.md) |
| The plane's log or the console's Overview says its OpenBao role "may not" do something | the role lacks that grant; the line comes with the bao lines an OpenBao administrator runs to add it (openbao.md) |
Worker pods stay in ContainerCreating or CreateContainerError naming an idmapped mount |
a volume's filesystem does not take idmapped mounts, as NFS does not (worker-sandbox.md) |
Worker pods stay in CreateContainerError, "cannot load seccomp profile" |
the Localhost profile is not on that node: kubectl -n troupe-system get daemonset troupe-worker-profiles, or where something else installs it, its path against worker.seccomp.localhostProfile |
Worker pods stay in CreateContainerError, "apparmor profile not found" |
the AppArmor profile is not loaded on that node: kubectl -n troupe-system get daemonset troupe-worker-profiles and the log of its apparmor container, or where something else loads it, its name against worker.appArmor.profile |
| Worker pods fail at once, "Cannot enforce AppArmor: AppArmor is not enabled on the host" | the node runs no AppArmor and the worker asks for a profile: worker.appArmor.enabled: false (worker-sandbox.md) |
A client is refused with update_required |
it is below the minimum the console's Client versions sets, or on a version blocked there (routine-tasks.md) |