Skip to content

A pod the operator removed is forgotten: a pod row whose substrate no longer runs it (an ordinal at or above the profile's count, a host no longer registered, a profile the plane no longer has) is listed gone, since it was last heard, once it is quiet past its lease, and the sweep drops the row once it is quiet past the drain deadline and holds no active session; a gone pod offers no action and is nowhere a drained pod's sessions could go

Decision 994 · accepted · 2026-11-15 · issue #883

What the plane decides

Listed is the substrate's word, not health (Fleet.listed?/2). For a profile the cluster makes, a pod is listed while its ordinal is below the profile's replicas, the count the plane writes to spec.replicas and the operator's StatefulSet runs; for a registered host, while a host of that name is registered for the profile; a pod of a profile the plane no longer has is listed nowhere. A pod that is listed and silent is broken, or starting, and stays the administrator's to look at.

Gone, then forgotten. A pod that is not listed and has not been heard from for longer than its lease is gone (Fleet.gone?/2), and admin.profiles.list says so on its row (gone: true, with last_seen_at as before). The sweeper's third pass, after marking pods unhealthy and rescuing what lost pods held, drops each row that is not listed, has been quiet for longer than the drain deadline (drain_timeout_seconds, five minutes by default, which Fleet.drain_deadline_ms/0 now reads for the drain too) and holds no active session (Fleet.forget_gone/0). The delete asks the heartbeat again, so a pod that enrolled between the read and the write keeps its row; a pod that comes back at the name later enrols afresh, as it always could. Sessions' worker_id is nilify_all, and a row holding an active session is never dropped, so nothing the rescue still has to reach is lost from under it.

Why the drain deadline. It is the longest the plane waits on a pod's running turns, and the one number already in the deployment that says how long a pod may be in the middle of leaving. A row dropped sooner could be one a drain is still waiting on; a row kept longer is only noise. One sweep, on the timer that already counts "last heard": no new process, no new column, no migration.

Not chosen: asking the operator for its pods. The plane has the count it wrote and the hosts it registered; asking the API server every sweep would answer "what exists" at a round trip each time, on every replica, for the one fact the plane already holds. Not chosen: dropping at once when the count comes down. The pod is still sealing and reporting then, and its row is what Drain reads; a gone row for a few minutes is the honest state.

What the console shows

A gone pod's row reads gone and when it was last heard, in muted grey, with no Drain; the region's count (ux · 1 of 1 pod) counts the pods still run. A drain's preview counts as somewhere the sessions could go only a pod of the profile that is healthy, not draining and not gone, whatever the gone row's other flags last said. The Overview leaves gone pods out of the fleet's count, of each profile's state and of "Needs attention": a removed pod is nobody's to look at, and a profile short of a pod is still said to be short. The fake plane seeds one gone pod (troupe-w-ux-1).

Governs: apps/troupe_plane/lib/troupe/plane/fleet.ex, apps/troupe_plane/lib/troupe/plane/fleet/sweeper.ex, apps/troupe_plane/lib/troupe/plane/admin.ex, clients/gui/apps/console/src/views/Pods.tsx, clients/gui/apps/console/src/views/Overview.tsx, clients/gui/packages/client/src/admin.ts