Architecture: the plane, the pods and what a session carries¶
How a session runs remotely (the plane, a worker pod, the operator and the admin surface), what a profile carries and the work nobody starts, what a session cost, provenance and entitlements, and what belongs to a person. The parts and where they live are the map; the harness a pod runs is the harness and its clients.
1. Remote¶
A client reaches the plane over HTTPS and a session's pod over a WebSocket of its own, pods reach the plane over the control channel, and the plane is in the path of neither a session's events nor its storage:
flowchart TB
client["troupe, the desktop app,<br/>or a browser at /app"]
idp["Identity provider"]
plane["troupe-plane<br/>/rpc · /auth/exchange · /mcp"]
op["troupe-operator"]
pod["worker pods, troupe-w-<profile><br/>/v1/socket"]
pg[("PostgreSQL")]
bao[("OpenBao")]
s3[("Object storage")]
model["Model provider<br/>or gateway"]
client -- "sign-in" --> idp
client -- "HTTPS: exchange, /rpc" --> plane
client -- "WebSocket, a token for that pod" --> pod
pod -- "control channel, TCP 4001" --> plane
plane -- "WorkerProfile, through the Kubernetes API" --> op
op -- "a namespace of pods per profile" --> pod
plane -- "index, identity, ledger, audit" --> pg
plane -- "transit: sign tokens" --> bao
pod -- "session keys" --> bao
pod -- "sealed segments" --> s3
pod -- "model calls" --> model
And the order a client does it in, from sign-in to a live session:
sequenceDiagram
participant C as Client
participant IdP as Identity provider
participant P as Plane
participant B as OpenBao
participant W as Worker pod
C->>P: GET /.well-known/troupe
C->>IdP: device grant, or PKCE in a browser
IdP-->>C: id_token, refresh_token
C->>P: POST /auth/exchange
P->>B: transit sign (ES256)
P-->>C: plane token (≤ 15 min)
C->>P: /rpc session.create
Note over P: row → Budget → Placement
P->>W: session.activate (control channel, epoch)
P-->>C: endpoint, token (aud = the pod)
C->>W: WebSocket initialize, subscribe
W-->>C: replay, then live events
W-->>C: auth.expiring
C->>P: token.mint
C->>W: auth.refresh (same socket)
1.1 The plane¶
The plane decides what may happen and where; it never sees what happens, and is never
in the data path of a live session. Identity: it is a plain OIDC relying party;
/auth/exchange verifies the provider's token and mints a plane token signed through
OpenBao's transit engine, so the plane holds no key it could leak (kid is the RFC 7638
thumbprint, so plane and workers agree with nothing kept in step).
The control channel. Workers dial an internal TCP listener, never exposed through
ingress, with their projected ServiceAccount token; a TokenReview validates it and the
namespace decides the profile, so a pod can only enrol as what it is. Upward go
heartbeats, the session index (ids, epochs, sequence numbers, head hashes), status
columns and usage batches; downward, idempotent pushes (activate, dormant, drain, erase,
fence, JWKS and ACL changes, config.updated), routed to whichever replica holds the pod.
Bundles are fetched, not pushed, and so are credentials, each for a session the pod holds
alone (one it does not hold gets not_found), the owner read off the session row:
kms.assertion {session_id}answers a short-lived JWT for that session owner's key slots, naming the owner's name at the key manager, under which the pod reads them.git.credential {session_id}answers a GitHub installation token the plane mints with its App for the repository on the session's row, narrowed to it and tocontents: read, with the URL to clone it from; the pod's own git uses it for the clone on the session's first activation, and the session never sees it (Decision 885).- With
access: "write"it answers the fix loop's token, with what the session's worker profile grants (spec.github.permissions). The pod's owngit_push,pr_open,pr_commentandpr_checksuse it outside the sandbox to push the session'stroupe/*branch through GitHub's API, open its draft pull request and read its CI, and report each step withgithub.record {session_id, ...}, metadata the plane checks, keeps on the session's row and audits (Decision 891).
The harness API (/rpc) is a fleet API: me, teams.list, profiles.list,
sessions.list, session.get, session.create, session.open, token.mint, archive,
pin, erase, session.grant, session.review, trigger.fire. session.create is a
sequence of reservations, each given back if a later one fails: row → budget (Budget:
the platform's, the team's and the person's ceilings) → capacity (Placement, or
pending while the profile grows) → push to the pod → token. The row comes first because
a placement is a conditional write against it, which survives the replica that made it.
Placement, TeamBudget, PersonBudget and PlatformBudget are :global actors (one per
profile, team and person, and one for the platform), which is why plane replicas must
cluster. session.open read never activates, since a session that woke because somebody
looked would never stay dormant; in activate mode a conditional epoch bump decides the
one caller that places it. sessions.list carries the status columns a pod reports
(status, done_reason, failed_reason, pending_approvals, pending_questions,
cost_micros), so a review queue is a listing and not a replay. The plane pushes nothing
to clients: /rpc is request and answer.
A laptop's session offered for steering is the one place the plane is in a session's
data path, because nothing can dial a laptop: its daemon dials /v1/relay with its
person's plane token, and the plane relays the protocol between that connection and each
person attached at /v1/relay/session with a token remote.open minted for that session
alone. Both ends hold every message to one table in the protocol app
(Troupe.Protocol.Relay), the laptop again to its own record of whom it shared the session
with, and each remote person is an ordinary connection on the laptop, so the session's rules
hold for them as for anybody. The plane keeps who offered what to whom and whether the
laptop is there, and no copy of what it relays (Decision 921).
Tokens for pods. aud is the pod's worker id: an audience naming the profile would
make pods interchangeable, which is what a leaked token wants. Workers verify offline
against a cached JWKS, warn with auth.expiring two minutes out and accept auth.refresh
on the open socket. The role in a token is a claim about the moment it was minted; access
is checked again on every command against the ACL mirror the plane pushes, so a revoked
collaborator is refused on their next command, and a narrowed one runs it with the
narrower role's scopes.
1.2 A worker pod¶
One process per active session, none per dormant one. Everything a dormant session is
lives in object storage, and activation is the only way back, which makes relocation and
losing a volume the same operation. Sealing: a sealer per active session uploads an
AES-256-GCM segment at every turn's end and at least every 60 s; that interval is the whole
durability promise. Upload, then report: a segment the plane knows of but storage lacks
would let a rebuild claim history it cannot produce. The same Troupe.Sessions.Sealer
runs on a pod and in a daemon, so either can restore the other's sessions.
Dormancy is: record, seal, stop the tree, archive and upload the workspace, report,
then erase the local copy. Epochs are minted by the plane alone; a pod whose epoch has
passed refuses to activate (checked against the plaintext manifest before decrypting), and
a fenced running session stops and discards its cache. Draining (scale-down, a new
image, an admin's drain) stops placement in the database and waits for running turns up to
the grace period, and the plane checks its own index before agreeing the pod is empty. A
scale-down lowers the count only past pods drained that way; a pod stopped by anything else
drains itself on SIGTERM. Disk pressure evicts caches, least recently used, never an
active workspace. Erasure is driven by the plane, which destroys the session's key
first, with the metadata delete its policy has for this and nothing on the data path,
so nothing under the prefix decrypts (not old object versions, not backups) and the
deletion after it is tidiness. A team session's pod is told at once: it stops the session,
drops its copy and deletes the objects, and an offline pod applies pending erasures when it
enrols. A private session has no pod: its objects go when the owner's daemon next connects
and says it has stopped (Decisions 756 and 811). A key the key manager refused leaves the session
erasure_pending, and the plane tries it again every five minutes (Decision 811).
1.3 The operator¶
| Resource | Scope | Written by | Says |
|---|---|---|---|
TroupePolicy |
cluster | a cluster admin, never the plane | what any profile may ask for |
WorkerProfile |
troupe-system |
the plane | what one pool of workers should be |
TeamVolume |
troupe-system |
— | that a team has shared storage |
spec.teams is a projection of the plane's grants, not a second source of truth.
Policy is checked twice, on purpose: a ValidatingAdmissionPolicy gives a person an
error the moment they ask; the operator's check still holds when admission is down or the
policy tightened afterwards. Resources.for_profile/3 is a pure function from profile,
policy and settings to manifests, so naming, addressing and egress are tested without a
cluster, and the reconciler only applies and compares: level-triggered and idempotent,
pruning by label (owner references cannot cross namespaces), led by a Lease. A pod gets a
projected token with audience troupe-plane, never the ServiceAccount's own, so it
cannot be replayed against the API server. StatefulSets are OnDelete because a pod holds
live sessions; the operator replaces a pod on an older revision once the plane, which alone
knows when a pod holds nothing, has drained it and recorded that on the profile.
Egress is default-deny: DNS, the plane's control port, OpenBao, object storage, the
model, the profile's MCP servers and git hosts. Plain NetworkPolicy cannot name a host, so
without Cilium the external ones are a wide rule on 443 and 80, with the installation's own
OpenBao and object storage and the profile's own endpoints beside it on the ports they
name, recorded rather than hidden. With Cilium the operator writes the toFQDNs rule the
profile asked for (with the installation's own OpenBao and object storage in it when they
are outside the cluster, a toCIDR of one address for a host given as an address), a DNS
rule through Cilium's proxy so it can learn addresses, and no wide rule beside it, since
Cilium admits the union of every policy on a pod. In either mode a profile's own endpoint
at a loopback or link-local address is admitted by no rule, and is refused where it is set
up and reported on the profile. Which mode a profile has is its EgressByHostname
condition, and the plane claims egress by hostname only where that condition says so.
1.4 The admin surface¶
Every administrative action goes through Troupe.Plane.Admin; the console, the admin.*
JSON-RPC methods and the MCP server at /mcp are renderings of it. The console is a client
of those methods and nothing else (the plane serves no administrative pages of its own), and
a parity test asserts every function has an API method and an MCP tool, so nothing the
console can do is out of reach of a program or a model. For a
model the method's summary and typed arguments are the interface, and a destructive tool
takes a confirm argument repeating the identifier. /mcp also accepts the identity
provider's own token, because an MCP client does OAuth against the provider and has no
step at which to get a plane token; Troupe is a resource server, never an authorization
server.
platform_admin is membership of an identity-provider group, never granted in Troupe,
because a role Troupe could grant would be a way to escalate inside it. team_admin is
assigned per team. Neither can read session content: no admin method returns events.
Break-glass is a separate, audited way into the console for a lock-out: a token exchanged
at POST /auth/breakglass for a short-lived plane token, with no more access.
Profiles are applied directly (server-side apply of the whole resource) or held by
GitOps: a repository holds the WorkerProfile, TroupePolicy and Trigger resources
and something else applies them; the plane never writes git. A cluster singleton lists the
profiles, then the triggers, every fifteen seconds and makes the plane's rows follow them:
a resource that fails the plane's own checks is reported rather than used, admin writes are
refused as managed_by_gitops, and the console is locked. The plane then writes only what
no repository could know, a profile's replicas, teams and mcpServers, under its own
field manager (Decision 736). A trigger's key, runs and revisions stay the plane's, and
running one by hand is not a write (Decision 737). Settings override the deployment and
never replace it: a stored row wins, reset deletes it, and a platform_admin_group cannot
be saved until a check passes for the value in the field. Audit is a row per change
with the actor and a diff keyed by path (spec.llm.model), computed by the same function
the editor previews with.
2. What a profile carries, and work nobody starts¶
Bundles (agent definitions, skills and MCP servers) are one versioned, hashed,
immutable document per version, validated at publish by the plane and again by the worker
after checking the hash. A worker keeps every version under bundles/<hash>/, so a session
pinned to a version reads that version however many have been published since; retiring it
moves pinned sessions on at their next activation. The bundle is the MCP servers' one
source of truth: pods discover tools on config.updated and apply its allowlist, and the
plane writes the servers into the WorkerProfile so the operator opens egress and injects
each credential from troupe-mcp-<name>.
A service principal (svc:<team>/<name>) is a credential a team owns, not a member,
which keeps membership the identity provider's business; it exchanges its secret for the
same plane token a person gets and has no admin role. session.create takes what an
unattended session needs: a first prompt the plane passes on and never stores; terms
(a budget slice, max_turns, wall_clock_seconds and approvals: wait | deny, with no
auto: a trigger that needs none runs a profile whose bundle sets those tools to auto,
an administrator's versioned act); and an origin. A trigger is a row the plane stores
and something fires: trigger.fire renders the template and creates the session as the
trigger's principal through the same call a person's client makes, and the same
idempotency key returns the same run. Cron triggers fire from a :global scheduler in the
plane; from outside, a trigger is fired at POST /trigger/<id> with a key of its own that
fires that trigger and nothing else, or by an executor that calls trigger.fire.
The A2A facade maps the A2A protocol onto sessions: a task is a session, a message is
input.send, a stream is a subscription, input-required is an approval answered by a
structured decision, artifacts are published events and blobs served with the caller's
own token. It holds no database and no credential of its own (docs/a2a.md).
3. What a session cost¶
Cost is a fold over the log, not a second thing to write down. Every llm_response
carries the model and, behind a gateway, its request id and cost: the id from a response
header (x-litellm-call-id), the cost from the usage.cost LiteLLM writes into the stream
when asked, which the OpenAI adapter does on every call to a gateway, or else from
x-litellm-response-cost (Decision 1065). Troupe keeps no price table of its own: the
gateway prices a call where it can, and a reconciliation compares the ledger with it by
request id. A gateway that names no cost leaves the harness to price the call itself, from
the provider's catalog or else from models.prices (a profile's llm.prices), and mark it
priced_locally (Decision 689). A call nobody priced
is recorded at zero cost and said once a session in the log; its usage record says
priced: false, and the plane counts such calls apart and says them on the profile and
team (Decision 1073). One with no request id gets a
synthetic id seq:<session>:<n> that the reconciliation counts as unmetered.
On a pod, Session.Log hands each event to a usage fold that writes an ETS row from the
log's own process (no mailbox on the turn path), drained in batches over the control
channel. The plane answers each batch with usage_seq, the highest sequence now recorded;
the pod drops what it held up to it and, after a restart, re-folds its log from it. The
watermark only moves forward and is deliberately not fenced on the epoch, because a fenced
pod still made the calls it reports. The ledger's uniqueness on request id turns any
overlap into duplicates rather than charges. There is no rollup pipeline: the raw table,
an index and a one-minute cache answer every question at this volume.
4. Provenance and entitlements¶
A trigger run names a revision: the trigger document frozen and addressed by the hash
of its canonical form, source inside the hash and enabled outside it. Editing a
template does not rewrite what old runs ran, and the session a run made carries the hash
in its origin. A grant may carry entitlements: which of a bundle's agents, skills and
MCP servers come with it; no rows is no restriction, deny wins. The bundle is never
altered: the plane resolves a set at create, profiles.list shows the union across a
person's teams, the session gets its one team's set, and the pod filters agents after the
whole search order is merged, skills after the definition's own list, and each session's
MCP tool list; the pod still discovers once.
5. What belongs to a person¶
Key paths take an owner: troupe/teams/<team>/sessions/<id> for a pod, scoped to its
granted teams, and troupe/people/<name>/… for a person, which no pod rule mentions. The
name is the person's at the key manager, a column on their row: their subject when they
were first known, left alone when a switched subject_claim moves their subject, so
nothing under it has to move (Decision 755). Whatever reaches the key manager for a person
takes the name from the plane's answer, never from a subject. A bundle's MCP server may use
credential_mode: person (the session owner's credential, fixed at activation), kept at
troupe/people/<name>/mcp/<slot>, which only that person can read: the pod exchanges a
plane-signed assertion at OpenBao's JWT auth for a token templated on the name, and a
person connects a server with me.connections.grant, which takes no value and returns
none. Where nobody has connected, the tool answers not_connected rather than a 401 the
model would retry.
A private session runs on a person's machine and never touches a pod. The daemon seals
it with the same sealer under the person's key subtree; the plane holds a row
(kind: private, no team, no profile) and nothing else. Bytes cross through presigned URLs
for one key under the session's prefix (session.presign), listing through
session.objects, and the epoch fences two devices waking the same session:
session.register with claim: true bumps it conditionally, and the loser learns at its
next seal, when the plane will not sign a write naming its epoch, so it writes nothing more
under the prefix and keeps its local log read-only (Decision 800).
The daemon's plane token comes from the client that signed in (identity.link), which
hands it over again when it renews it and when the daemon has restarted; it is held in
memory only. Each link that carries one is also when the daemon asks what was erased while
it was away (session.erasures), drops its sealer and its copy of each, and says so
(session.erased), which is when the plane deletes the objects; and when it carries on
sealing each private session it has no sealer for, from the row's last_seq at the epoch
it held, or from the first event for one the plane has never heard of (Decision 764).
Signing out at the client takes the token back (identity.sign_out), and the daemon stops
every sealer until the next link carries them on (Decision 766).
6. Supervision trees¶
Worker (Troupe.Worker.Supervisor, empty unless TROUPE_WORKER_AUTOSTART):
KMS.OpenBao.Login first (the worker's own OpenBao identity), Sessions (a
Session.Manager per active session), Auth, MCP.Sessions, Connections,
ClientCredentials, MCP,
Bundles (before the link: enrolment claims the bundle hash), Usage, Disk.Watch,
Plane.Link (only with
TROUPE_PLANE_CONTROL), and Harness — the gateway's Commands, Connections, a raw
NDJSON Listener on 4100 and Gateway.Web (Bandit on 4000: /health/live,
/health/ready, /v1/socket).
Plane (Troupe.Plane.Supervisor, empty unless TROUPE_PLANE_AUTOSTART): Repo,
Settings (5 s cache), PubSub, Singleton (a DynamicSupervisor for the :global
actors: one Placement per profile, one TeamBudget per team, one PersonBudget per
person, one PlatformBudget, and the fleet scaler, GitOps and erasure-retry singletons),
Fleet.Sweeper, the keepers of the trigger scheduler, the fleet scaler, GitOps
(Gitops.Keeper) and the erasure retry (Erasure.Retry.Keeper), Erasure.Tasks, Fleet.ReleaseImage (rewrites
release profiles after an upgrade), the control channel's registry, connections and :gen_tcp
listener on 4001 (plaintext or TLS, told apart by the first byte), Ledger.Cache, Tokens.Credential (the OpenBao login), the Phoenix
endpoint on 4000, and libcluster when RELEASE_DISTRIBUTION=name. The :global actors
are why the chart refuses more than one replica without distribution.
Operator (rest_for_one, under TROUPE_OPERATOR_AUTOSTART): Reconcilers (one per
resource), Watch (Bonny: watch, resync, Lease leadership), Descendants. A2A:
a plane-token cache, the stream registry, and Bandit when autostarted.
7. Things a reader will trip over¶
- The operator and the worker both compose a pod's hostname
<ordinal>-<profile>.<domain>; the two cannot share a function across the boundary, and both say so. troupe_planereads what it needs of the design (the front page's theme, whichmix troupe.themewrites topriv/static/theme.css) frompriv/, not fromclients/gui/docs/design/themes/directly: the image build does not seedocs/.