troupe bench measures the harness offline against budgets kept in one file, and CI fails past one¶
Decision 772 · accepted · 2026-10-04 · issue #390
The harness's cost curve (#389) was found by a person reading a bill: nothing in the
repository said how many model calls a turn makes or what each one sends, so nothing could
notice when that got worse.
- Shape, not answers. Five scenarios (Troupe.Bench.Scenarios): one turn of thirty
tool calls (calls per turn, the system prompt and tool definitions, the largest
prompt, growth per round trip, input tokens over the turn, events logged), a tool
result over the limit cut and read back with read_output, compaction at the
configured share of the window (never before, within one round trip after), a
cancel during a model call (its task ends, no call follows, the agent rests and
answers again), and the log against the session (the file folds as the running log
does, hashes chain, a replay from the start sees each event once, a resumed agent
has its conversation). Whether a model's answer is good is #114's question.
- A scripted model that counts. Troupe.Bench.Model answers the fake provider
as Troupe.LLM.Fake does, so the session runs the code a real one does, and
reports four bytes of the prompt as a token: the fake's flat 100 would never fire
compaction where a model's count would. Prompts are measured as the log writes them,
the scratch workspace's path written <workspace>, and tool calls are numbered
from one, so the JSON of two runs of one build is identical.
- Budgets in one file, compiled in. apps/troupe_core/priv/bench/budgets.json, a
maximum per scenario and measure, one file to review rather than one beside each
scenario; compiled into the harness, so an installed troupe bench holds itself
to what CI did. A budget naming a measure nothing takes fails. Moving one is a
change to that file, with both numbers in the pull request
(docs/developer/bench.md).
- This build's harness, in this VM, isolated. Each scenario has its own
workspace, config and state directories, and while it runs TROUPE_CONFIG_HOME
and TROUPE_STATE_HOME name them and opencode's files name none, so nothing of the
person's reaches the prompt being measured. The session is the harness's own in
the same VM rather than the machine's daemon, which could be another version and
would answer for itself; a daemon per scenario would measure a VM booting.
- One report for both tiers. Schema 1, JSON with sorted keys and a Markdown
table from it. Each scenario has metrics (value, budget, passed), checks, an
outcome a script checks whoever answered (a file's content, or a command exiting
0), and runs[], each the record the live tier asks for: model, outcome, stop
reason, turns, model calls, input, cached and output tokens, cost, wall clock,
retries, compactions, approvals, tools by name with failures and time, and every
call's prompt breakdown with its latency and first token. Offline, what needs a
clock, a price or a provider's own retries is null. troupe bench --live is
reserved and refuses: the live runner adds a runner, not a format.
- Where it runs. troupe bench [--json] (TUI Decision 140), mix troupe.bench
[--json PATH] from a checkout, a step of scripts/ci, and CI's lint job, which
puts the table in the run's summary and keeps the JSON. A few seconds.
- Reads the requests, not yet the log's breakdown. Offline the bench measures each
request the model is handed. When #389's first slice records each call's prompt
breakdown and each turn's sum in the log, a live run reads the same calls[] from
there.
Governs: apps/troupe_core/lib/mix/tasks/troupe.bench.ex, apps/troupe_core/lib/troupe/bench.ex, apps/troupe_core/lib/troupe/bench/history.ex, apps/troupe_core/lib/troupe/bench/runner.ex, apps/troupe_core/lib/troupe/bench/scenarios.ex