Testing¶
1. Running¶
mix test # every app, from the root
mix test apps/troupe_gateway/test/troupe/gateway/python_client_test.exs
cd apps/troupe_plane && mix test # one app
Tests live in apps/<app>/test/troupe/**/*_test.exs, with case templates and stubs in
test/support/. The test environment registers four misbehaving tools (raising, exiting,
asking, counting) through the same extension point an MCP adapter uses, and gives the
plane's sandboxed repo a pool of 30 because a placement test makes fifty creates at once.
A test waits for something another process does with eventually/1,2,3: the umbrella's
in apps/troupe_protocol/test/support/eventually.ex, which every app's test run compiles,
the TUI's in Troupe.TestHelpers. A suite writes no poller of its own.
CI runs each app's suite as its own job; the nightly and a release run every suite nine
more times, because the suite is concurrent and a race that shows one time in five is a
bug this project cares about. The nightly also runs the TUI's suite three more times and
the GUI's five, in a random order, and names every test that failed in any run
(scripts/flake-hunt, which runs any suite that way: scripts/flake-hunt 20 mix test <file>).
2. What each suite needs¶
The policy is to skip loudly: a SKIPPED: block naming the command that brings the
dependency up, never silence.
| Dependency | Suites | Without it |
|---|---|---|
| PostgreSQL on 25432 (the suite creates and migrates its checkout's own database) | all of the plane's | every test excluded, with the two commands to run |
| PostgreSQL | the worker's control-channel failover test | skipped, flunks |
| OpenBao on 28200 | plane tokens, protocol KMS, every Troupe.Worker.SessionCase suite, the gateway's PrivateTest sealing |
setup_all says so; each test flunks (the plane's PersonCredentialsTest and PrivateErasureTest, which need MinIO too, are skipped by a tag instead) |
| MinIO on 29000 | protocol object store and storage, worker sessions, the gateway's PrivateTest sealing |
same |
| a kubeconfig with the CRDs | the operator's :cluster tests, the plane's enrolment test |
excluded; --include cluster runs them |
| a second BEAM node | the plane's cluster test | skipped, flunks |
bubblewrap |
the sandbox tests | skipped, flunks |
python3 |
the conformance test | skipped by a tag, saying so |
inotifywait / mac_listener |
the native half of the watcher tests; two gateway FilesTest tests (an event inside a second, a deletion) |
watcher: passes without asserting, polling is always tested; FilesTest: skipped, saying so, except under CI=true, where they fail |
zig |
anything that runs shell |
:reaper_missing |
scripts/dev-up provides the first four; local-setup.md. Suites isolate
themselves from the developer's machine: the core points TROUPE_CONFIG_HOME at a
temporary directory once per run and passes state_dir through config, because
System.put_env/2 is global; worker tests set TROUPE_STATE_HOME per test and are
async: false. Suites assert on events and telemetry, never on log lines.
3. What keeps a test from flaking¶
Decision 915 is the long form.
- A GUI test waits for what comes after an async step:
await sees("…")(and the console'sawait seesLink("…")), notexpect(says("…")).toBe(true), which each app'swaits.test.tsxrefuses unless the line says why it is meant at once (// at once: <why>). A control is the same:await seesButton("…")(the console'sawait seesField("…")too, orawait waitFor(() => row(…), …)) and read it from what the wait answers,expect((await seesButton("Save")).disabled).toBe(true), notexpect(button("…")).not.toBeNull()orbutton("…")!.disabled, which the same check refuses forbutton,fieldandrow. Absence (toBeNull()) is not checked: it cannot flake this way, only pass early. To show a race on purpose, make the fake slow:daemon.delayMs["config.get"] = 300, orsubscribeDelayMs. - An async ExUnit module sets no environment variable but a
TROUPE_TEST_...one of its own;Troupe.Credo.AsyncEnvchecks it. A test that needs another home isasync: falseor passes the home as an option. - Each checkout has its own plane test database,
troupe_plane_test_<hash of its path>, so two worktrees' plane suites no longer deadlock (40P01); CI keepstroupe_plane_test, andTROUPE_PLANE_TEST_DBnames one. - The GUI's client tests read nothing from outside
clients/guithatclients/gui/Dockerfiledoes not copy and the root's.dockerignorekeeps: the image runs them with that context alone (packages/client/test/support/image-reads.ts). - A flaky test is fixed at its cause: no retry, no longer timeout, no skip.
4. Fixtures¶
test/fixtures/logs/<version>/— six recorded logs per recorded version (0.2.0 and 0.3.0; no release since has recorded its own) and the fold hash each produces.mix troupe.fixtures.record <version>writes them and refuses to overwrite.fold_test.exsreplays every version through the upcaster and compares hashes, and readsagent/server.exto assert every event type the replay handles is witnessed. A moving hash is not a test to update.fixtures/sample_repo/— a small Mix project the core's workspace and tool suites read.apps/troupe_gateway/test/conformance/—troupe.pyandconformance.py.- Fakes: a pod on the control channel, an enrolment verifier with no API server and a
second plane node (plane); a stub plane and a fake worker (A2A);
WorkerProfileandTroupePolicymaps as the API server hands them over, andFakeCluster, an API server in memory that a reconcile pass runs against (operator).
5. Checks that are tests in all but name¶
- The Python conformance client is a client written against
PROTOCOL.mdin the standard library alone. The test starts a daemon on a Unix socket with the fake provider, runsconformance.pyagainst it, and asserts on its report: contiguous replay fromseq1, an acceptedinput.send, an answered approval, a verified hash chain. It is what proves a client needs no private access. - The admin parity test asserts every public function of
Troupe.Plane.Adminhas an API method and an MCP tool, every method a summary and described arguments, every destructive one aconfirm, and that nothing returns session content. - The schema tests: a real session's every event validates against
Troupe.Protocol.Schema, and the compatibility rulesmix troupe.schema.diffenforces. - The asset tests: every asset the console's and the front page's documents name exists, is allowlisted and is served, and the front page spends the reserved colour once.
- The egress test:
docs/egress-allowlist.mdnames every host the components declare. mix troupe.boundariesis part ofmix check, notmix test.mix troupe.bench,troupe benchfrom a checkout: the harness's offline suite, each number held to a budget inapps/troupe_core/priv/bench/budgets.json; CI'slintjob andscripts/cifail past one (bench.md).
6. Beyond the unit suites¶
mix troupe.e2e(operator): the cluster suite, againstscripts/remote-up's kind cluster; CI'sclusterjob.scripts/kind-upgrade-check [--from <version>] [--recreate]: the release before this one from the registry, an organisation seeded through the protocol, this tree's CRDs andhelm upgrade, and every seeded thing asked for again; CI'supgradejob, and what a fixer runs under the plane lock before pushing a change to the chart, the operator or a migration (ci.md).scripts/kind-upgrade-check require-tls [chart]is the other upgrade it checks, againstscripts/remote-up's cluster: a plane without TLS on its control port, a pod made then, and the upgrade torequireTls: true, which passes when the operator replaces the plaintext pod by itself.scripts/pitr-drill: restores the compose PostgreSQL to a point, rebuilds the index and asserts the lost session came back (../admin/backup-restore.md).scripts/dev-restore-rehearsal [--keep]: a database and a bucket of its own on the dev stack, seeded through the plane's modules, dumped (pg_dump,mc mirror), dropped, restored, and asked whether every table, the audit chain, the memory, an object's bytes and the index rebuilt from the bucket came back; prints each step's time. Run wherescripts/dev-upandmixrun (on Windows, under WSL). The restore the admin pages outline is this one (../admin/upgrading.md).- The TUI's suite (
mix checkinclients/tui) and the GUI's (pnpm test), whosetest/supportis a protocol-accurate identity provider, plane, worker and daemon; the GUI's plane end-to-end suite runs in CI against a plane built from the same commit (e2e.md). - The VS Code extension's:
pnpm testinclients/vscode(the folder, the program, the line it types, put through every shell the machine has, the Settings view's rows, its Models group and the manifest), andpnpm test:vscode, its suite inside a real VS Code against a faketroupe, with its own user data and extensions directories (its README). scripts/verify-local.ps1checks an install made byscripts/install-local.ps1on this Windows machine.scripts/check-installers(install.shundershandbash, or the shells named) andscripts/check-installers.ps1(install.ps1, run once withpowershelland once withpwsh): the installers against a release of stand-ins in a scratch directory, whosetroupe-daemononly writes down what it is asked. Nothing is downloaded, and no real install,PATHor login entry is touched. The questions are answered through a pseudo-terminal forinstall.sh;install.ps1's are checked by lifting the function that asks out of the script, because a check has no console.scripts/live-check task|delegate|loop: one headless run against a real model, and what it has to show; the nightly runs all three against the gateway (ci.md). WithTROUPE_PROVIDER=fakeand aTROUPE_FAKE_SCRIPTit runs offline, which is how a change to it is tried first (a step with"delay_ms"sends nothing for that long, which is how a client's waiting state and the harness'sllm_first_byte_msare tried offline;Troupe.LLM.Fakelists the steps). Its cost column is the gateway's price, else the oneTROUPE_MODEL_PRICESgives (priced here); a script with"cost_micros": nullanswers as a gateway streaming a response does, with no price, which is how that column is tried offline.