A scale-down drains the pods it removes before it lowers the count, and finishes once it has started; a worker stopped by anything else drains itself¶
Decision 731 · accepted · 2026-09-29 · issue #273
Issue #273.
The scaler wrote a lower spec.replicas as soon as the grace period had passed, and
the StatefulSet took the highest ordinals with what they held: a turn in flight and
the workspace since its last archive, while clients saw the session vanish until the
sweeper marked it dormant from its last seal. Drain.scale_down/3, which the
scaler's documentation relied on, had no caller but a test, and is gone. Now the
scaler marks the pods above the count it wants retiring and draining, drains them
in the background with the drain an administrator starts (Drain.start/2, which the
upgrade of 726 uses too), and lowers the count from the top past each retiring pod
that is draining and holds no active session on a count taken before that tick,
never past one the plane still counts a session on. A turn in flight gets the drain
timeout, the number the grace period is set from, and is then cancelled with
everything before it sealed: bounded, because a session that never rests would
otherwise keep a pod nobody needs, and no harsher than deleting the pod. Nothing
retries; the pod is asked once and each tick reads the count. A scale-down that has
started is finished even when the sessions come back, because a drained pod takes no
session until it restarts (633) and kept it would be room counted that is not there:
the count comes down past it and the next tick asks for a fresh pod. retiring is
what tells this drain from an administrator's, which stays a person's (633, 726)
unless its pod is above the wanted count; it is cleared when the count passes the
pod and whenever a pod is not draining. 726's troupe.dev/drained record is not
used: it tells the operator to replace a pod behind, and nobody but the plane is
needed to remove a pod, since it writes the count. A count that grows while a
retiring pod still drains grows past it, and that pod goes with the next scale-down
that reaches it. The worker drains on SIGTERM as well (prep_stop/1) for pods the
plane did not stop: turns get half its drain timeout and the rest of the grace
period is for sealing, archiving and reporting; a shorter grace period ends it as a
stop always did. The plane migrates (workers.retiring). Proof: the plane's
scaling_test.exs (drains first, a busy pod delays it, a drained pod goes, a started
scale-down finishes, an administrator's drain is left), the worker's drain_test.exs
(a pod stopped mid-turn puts its session to sleep and its files survive the volume),
and the cluster suite's capacity_test.exs.
Governs: apps/troupe_plane/lib/troupe/plane/fleet.ex, apps/troupe_plane/lib/troupe/plane/fleet/scale_down.ex, apps/troupe_plane/lib/troupe/plane/fleet/scaler.ex, apps/troupe_plane/lib/troupe/plane/fleet/worker.ex, apps/troupe_plane/priv/repo/migrations/20260929000034_worker_retiring.exs, apps/troupe_worker/lib/troupe/worker/drain.ex