Skip to content

Backup and restore

What state exists, which copy is authoritative, what the repository gives you to get it back, and what it does not.

1. Where state lives

Store Holds Rebuildable?
Object storage sessions/<id>/… sealed event segments (AES-256-GCM per session key), plaintext manifests, snapshots, workspace archives, blobs — the session truth no; this is the primary copy
PostgreSQL the session index, ACLs, tombstones; the ledger; the audit trail; the identity mirror (people, groups, teams, grants, team admins); profiles and workers; bundles; principals, triggers, runs, settings; the client version policy the session index yes, from manifests. Everything else exists only here
OpenBao per-session data keys (KV v2), a team's and a person's private ones; people's own MCP credentials (troupe/people/<name>/); the transit signing key, and the keys profiles sign MCP assertions with no. Losing the KV keys makes every sealed segment unreadable forever; losing the transit key invalidates every outstanding token; losing the rest means people and profiles sign in to their MCP servers again
Worker volumes /var/lib/troupe working copies of live sessions, caches, materialised bundles, the unsealed tail of the log — at most 60 s or one root turn yes, except that tail
Kubernetes WorkerProfile, TeamVolume, TroupePolicy, worker namespaces in direct mode, profiles from the plane's rows (re-apply); the policy from your values; the rest by reconcile
A repository (gitops mode) the WorkerProfile and Trigger manifests, and the TroupePolicy or the values that make it the primary copy: the applier puts them back in the cluster, and the plane reads its profiles from there at its first pass (gitops.md). Grants, teams and the rest of the plane's rows are still only in PostgreSQL

So a restored database plus object storage is a complete plane for sessions, and the database backup is the only copy of principals, settings, the audit trail, and triggers (in gitops mode their definitions are the repository's, and their keys and runs still only the database's).

2. What the repository provides

scripts/pitr-drill is a restore drill against the docker-compose development PostgreSQL only (which has WAL archiving on for it). It takes a base backup and a named restore point, writes a session after it, restores to the point as a second cluster, asserts the late row is gone, rebuilds the index from object storage, and asserts the session is back with the right epoch and head hash. Run scripts/dev-up, then scripts/pitr-drill (TROUPE_PG_CONTAINER and TROUPE_PITR_DB override the container and database). It is not a disaster-recovery tool, and nothing schedules it against a real instance.

scripts/dev-restore-rehearsal is the restore of a database and a bucket rehearsed on the same development stack, with each step timed: a database and a bucket of its own, seeded through the plane's modules, dumped with pg_dump and copied with mc mirror, dropped, restored, and checked table by table, with the audit chain, the organisation's memory, an object's bytes and an index rebuilt from the restored bucket. The restore in upgrading.md §7 follows its shape. A bucket copied that way has its current objects and not their earlier versions.

mix troupe.index.rebuild [--database-url URL] reconstructs the session index from object storage alone, with no key: manifests are plaintext, and every segment's epoch, last sequence and head hash are in its key and metadata. It is safe on a populated index, keeps a tombstoned session erased, and prints found, rebuilt, skipped, failed. A release has no Mix, so in a cluster:

kubectl -n troupe-system exec deploy/troupe-plane -- /app/bin/troupe_plane eval 'IO.inspect(Troupe.Plane.Index.rebuild())'

mix troupe.ledger.reconcile [--days N] [--database-url URL] compares usage with the gateway and never repairs (integrations.md §5).

Erasure is ordered so a restore cannot bring a session back: a tombstone first (the session goes read-only), then the plane destroys the key's metadata, every version, and then the objects go (routine-tasks.md says by whom and when). Because the key goes first, old object versions and backup copies are unreadable. That holds only with bucket versioning on, and only once the key is gone: a session the key manager refused is erasure_pending until a retry destroys it (Decision 811).

3. What it does not provide

No scheduled backup of any kind (use the managed database's PITR); no OpenBao Raft snapshots or backup of the unseal share; no object-storage lifecycle or replication rules; no scheduled ledger reconcile; no job that acts on a team's erase_after_days.

4. Restore procedures

The plane's database

  1. Restore the managed instance to the last point before the incident as a new instance, as the drill does.
  2. kubectl -n troupe-system scale deployment/troupe-plane --replicas=0. Live sessions do not notice — the plane is not in their data path — but nothing can be opened or created.
  3. Point troupe-plane-database at the new instance (kubectl create secret … --dry-run=client -o yaml | kubectl apply -f -) and scale back up, or helm upgrade if the schema needs migrating.
  4. Rebuild the index (§2), with object-store credentials in troupe-system.
  5. Workers re-enrol by themselves and receive pending erasures and the JWKS; usage they hold is re-sent from the plane's watermark, and the ledger's uniqueness on request id makes overlap a duplicate rather than a double charge.
  6. Accept the loss of what lived only in the database after the restore point — audit rows, runs, principals created or rotated, settings, client versions, team admins — and re-create what matters.
  7. Verify with admin.overview, admin.sessions.list, and a dormant session opened from a client.

Object storage

No recovery: after dormancy sealed segments exist nowhere else. Versioning protects against overwrite and deletion of single objects; nothing here configures replication.

OpenBao

If the Raft volume survives, the pod restarts and the sidecar unseals it. If it is gone, so are the keys and every sealed session is unreadable: re-create the transit key and the auth roles so new sessions work, erase the unreadable sessions so the index matches, and from then on take bao operator raft snapshot save on a schedule and keep the unseal share outside the cluster.

A worker volume

Delete the pod and its claim, data-troupe-w-<profile>-<ordinal> (the pod alone comes back on the same volume); it comes back with a fresh one, enrols, and its sessions restore from object storage at their next activation. Lost: at most the last 60 s or current root turn of a session that was live there, and the local blob store.

A bad migration

The failed hook Job is kept with its logs. To go back one schema version, run the plane image with the database URL, TROUPE_PLANE_AUTOSTART=false and RELEASE_DISTRIBUTION=none (and ERL_FLAGS="+Q 65536"), evaluating Troupe.Plane.Release.rollback(Troupe.Plane.Repo, <version>).