Skip to content

A worker upgrade finishes by itself: the plane drains a pod behind once it holds no active session and records that on the profile, and the operator then deletes it

Decision 726 · accepted · 2026-09-29 · issue #258

Issue #258. A worker StatefulSet rolls OnDelete, and since #253 UpgradePending names the pods behind, but nothing replaced them. The operator cannot tell a drained pod from a draining one, since readiness fails as a drain starts, and only the plane knows when a pod holds nothing; the plane has no pod delete and gets none (#253's option (c)). So each writes what it alone knows, where it already writes: the operator lists the pods behind in the status it owns (status.podsBehind, name, uid, revision), and the plane its finished drains in an annotation (troupe.dev/drained, pod to revision), because its grant is patch on the resource and not on the status, under a field manager of its own so its profile writes do not remove it. The plane, on the scaler's tick, drains the highest-ordinal pod behind that holds no active session, only while no other pod behind is draining, and records a pod once it is draining and a count taken on a later tick finds it empty. The operator deletes a pod that is behind, recorded at the revision it still runs, and not Ready, one at a time, highest ordinal first, and never while one is terminating or missing; not Ready is what keeps a pod that restarted after its record, and may have been given a session, from being deleted. The operator's list carries each pod's uid, and the plane keeps a worker's uid from its enrolment token's pod-uid claim, because a pod's replacement has the same name and can enrol before the next pass: drained by name, it would be a current pod drained for good. Placement puts a new session on a current pod while one has room (workers.upgrade_pending), so a busy pod's sessions go dormant in their own time. A profile with one pod gives it new sessions until every one is dormant, then is without a pod while the replacement starts. This is the "removes a drained pod" that Decision 633 still owed, for pods behind only: a drained current pod still waits for a person. What it costs: an idle pod's drain is started without waiting for it, and a drain whose answer never arrived leaves the pod Ready and recorded until somebody drains it again; a CRD without podsBehind drops it and the roll stays manual. Proof: reconciler_test.exs (behind and drained is deleted; not drained, Ready again or drained on another revision is kept; one at a time; current never), the plane's upgrade_test.exs and placement_test.exs, and the cluster suite's upgrade_test.exs, which rolls a one-pod profile with nobody deleting anything.

Governs: apps/troupe_operator/lib/troupe/operator/reconciler.ex, apps/troupe_plane/lib/troupe/plane/enrolment.ex, apps/troupe_plane/lib/troupe/plane/fleet/upgrade.ex, apps/troupe_plane/lib/troupe/plane/fleet/worker.ex