The cluster suite proves N-1 to N on every pull request into main that it runs on, and on every release¶
Decision 936 · accepted · 2026-10-24 · issue #765
The cluster suite (Decision 889) installed the chart fresh and ran the e2e; nothing anywhere installed the release before and upgraded it, so a migration that needs a hand step, a CRD change that breaks an old object, or a row the new code misreads was found by the first deployment to upgrade.
- What runs.
cluster.ymlhas a second job,upgrade, besidekind, on a runner and a kind cluster of its own; one after the other would be about an hour. It runsscripts/kind-upgrade-check, which a developer runs the same way under the plane lock. - Which release. The previous release is the newest
v*tag at or belowVERSIONthat does not point at the commit under test: on a chunk branch the release it was cut from, on a release commit the one before, and on a re-run of a release after its tag exists still the one before. Pre-releases order as releases do (1.0.0-rc.1before1.0.0).--fromorTROUPE_UPGRADE_FROMnames another,TROUPE_UPGRADE_CHARTanother chart source. - As a deployment has it. The previous release's chart from
oci://ghcr.io/it-minds/charts/troupe, its CRDs (helm show crds | kubectl apply --server-side), its images fromghcr.io/it-minds, and its own tag'sdev/kind/values.yaml, throughscripts/remote-up, which now takes a chart, a version, a values file and pulled images (TROUPE_CHART,TROUPE_CHART_VERSION,TROUPE_KIND_VALUES,TROUPE_IMAGES=pull,TROUPE_IMAGE_TAGfor every component). The services beside the chart indev/kind/dependencies.yamlare the tree's under both releases, because a deployment does not upgrade its database or its OpenBao with Troupe; where they changed since the previous release the script says so, as a precondition the upgrade page must name. - The seed, through the protocol.
apps/troupe_operator/test/upgrade/seed_test.exs, on the previous release: a team with a profile granted, the profile following the release (image: release) with the scriptedfakemodel so a turn needs no model; a session that ran a turn and went dormant; the organisation's memory; a client-version policy and the signing key a client pins; a webhook trigger with its key, fired once; and the audit rows that wrote. What it made is written to.e2e/upgrade/seed.json. - The upgrade, and nothing else. The tree's images built and loaded, the tree's CRDs
applied server-side,
helm upgradetocharts/troupewith its migrate hook, the rollout. There is no step for a person; one needed is a failure of the checks after it. -
The check.
test/upgrade/check_test.exs:/.well-known/troupereports the commit under test; the audit chain verifies and still has the seeded rows; the team and profile are there; the client policy is served and verifies under the key pinned before; the profile is Ready with its pods replaced on the new release by themselves; the dormant session wakes there with its old log read back and answers a turn; the memory reaches a new session; the trigger fires on its old key. -
When. On Decision 889's paths, plus
scripts/kind-upgrade-check, the plane's migrations andVERSION, which are added to the oneclusterfilter inci.ymlanddev-check.yml, so a migration also runs the fresh suite; in the nightly; and on every release, which runscluster.ymlin full. - N-1 to N only. A release skipped on the way is tested nowhere, and
docs/admin/upgrading.mdsays so: one release at a time, the backup, the CRDs, the chart and its migration, what to watch, what the plane checks,helm rollbackand what the migrations allow (no test rolls back), and the restore. - The restore, rehearsed.
scripts/dev-restore-rehearsaltakes a database and a bucket of its own on the dev stack (never the suites'), seeds them through the plane's modules, dumps them (pg_dump -Fc,mc mirror) off the servers, drops both, restores both, and checks every table's rows, the audit chain, the memory in force, an object's bytes and an index rebuilt from the restored bucket, printing each step's time. It is the shape the upgrade page's restore follows; it reads no kubeconfig.
Governs: .github/workflows/cluster.yml, .github/workflows/ci.yml, .github/workflows/dev-check.yml, scripts/kind-upgrade-check, scripts/remote-up, scripts/e2e, scripts/dev-restore-rehearsal, scripts/dev-restore-rehearsal.exs