Skip to content

troupe bench --live runs small tasks against the person's own model, under a cap it says before it starts, and keeps a history --compare reads

Decision 773 · accepted · 2026-10-04 · issue #390

The offline suite says what the harness does with a scripted model; what a turn costs on a person's own gateway and model, and whether the work gets done there, only a real model can say. - Tasks a script checks, not the offline scenarios. Troupe.Bench.LiveScenarios: write a file with given content; fix a failing test without touching it (elixir calc_test.exs exits 0); delegate a question to explore and write its answer; read a file that is never there and fall back to a default. Each has an outcome (Decision 772's {:file, ...} or {:command, ...}) and a check that the work was done the way the task is about (it delegated; the test is as it was; the read failed). The offline scenarios stay offline: their drives and measures read the scripted model's requests, their budgets are the script's, and thirty reads a turn against a real model is the expensive way to learn what the offline suite already says. A scenario whose outcome is a command not on the PATH is left out before anything is spent, and the plan says so. - A file's outcome is its text. {:file, path, content} now compares with line endings and trailing whitespace aside: a model asked for one line may or may not end it with a newline, and on Windows may write \r\n. The offline replay outcome holds as before. - A command outcome runs on the person's PATH. The VM puts its own runtime's bin first on the PATH its children get; in a release (the installed troupe, troupe-daemon) that runtime has no boot file of its own, so elixir calc_test.exs started on it and died at boot (cannot get bootfile ...start.boot), and the first installed run failed every fix_test the model had fixed. In a release the outcome command gets the PATH without that directory (Scenario.command_path/2). - The person's provider, in memory. plan/1 resolves the person's configuration and copies where a model call goes (provider, URL, key, named providers, windows, prices, the catalog) into each run's config overrides, nothing else of theirs; --model picks another model, addressed as troupe addresses one, which is also how another provider is picked, so there is no --provider. The key is never written to a file (a session's config is not logged), never printed, and inspect/1 of a plan leaves it out. A model whose key cannot be read is refused before anything starts. - A cap, said first, and asked about. Each run's limits are scripts/live-check's (12 model calls, 200,000 tokens sent, 24,000 received, 240 s), well above what any of the tasks needs and far below the defaults. The cap is those limits at the model's price (the catalog, else models.prices; the input at the dearer of the input and cache-write prices, since the budget counts both) times the runs, which is the issue's "budget x scenarios x repeats"; a model nothing prices has its cap in tokens. It is printed on standard error, and nothing runs without a y at a terminal or --yes; with no terminal to ask in, nothing runs (TUI Decision 141). While a run goes, the harness holds it to its limits and the runner stops it at its share of the cap, counted from each call's gateway.cost_micros, which also covers the cache reads the budget leaves out, so a run passes its share by at most the call in flight. - Each run isolated, and stopped whatever ends it. A run is the offline suite's kind: its own workspace, config and state directories with the environment naming them, and a session of the harness this binary carries, with every tool allowed and a budget that stops rather than asks. Not a daemon of its own: in this VM the session's tree is what scripts/live-check's daemon is there for, everything the run started, and stop_session/1 takes it all down when the root agent's turn ends, the run reaches its share of the cap, or its wall clock passes by a quarter more (at most 30 s), which catches a call that hangs between the harness's own checks. A daemon for each run would measure a VM booting, and could be another build than the one the bench ships in. Every tool allowed includes the shell, in a scratch directory that is not a sandbox; the plan says so, and the user page says to run it as one would such a session. - Measured as it happens. The runner subscribes to the session and stamps each event as it arrives: a call's latency is its llm_request to its llm_response, its time to first token the first llm_delta between them (an agent's calls are one after another, so they pair up by agent), a tool call's time its tool_call_started to its tool_call_completed, by agent and id, since a provider that gives no ids gets the same call_0 in every agent. Usage and cost are the log's. Retries happen inside the provider's call and never reach the log, so Troupe.LLM.Provider now says each as [:troupe, :llm, :retry] telemetry, which the runner counts for its run. What a call's prompt was made of is its llm_request.prompt_bytes (Decision 769), the offline suite's measure, and a compaction's summary is a call too, from the compacted that carries it, so its cost counts against the run's share of the cap. - One report, with mode: "live". Schema 1 (Decision 772) with the live fields filled, and added: the report's model, repeat, cap_micros, started_at and skipped; a run's error (why it did not end by itself), checks and succeeded (no error, the outcome held, every check held); a call's agent, cached_tokens and cost_micros. A scenario's metrics are taken over its runs and held to no budget: the success rate, median and worst cost, and the medians of the wall clock, a call's latency and time to first token, the calls and the tokens each way; a model's variance is not a regression. A scenario passes when every run succeeded, and troupe bench --live exits 0 then, 1 otherwise, 2 when it ran nothing. - History and --compare. Each run is appended to <state>/bench/results.jsonl as it ends (so Ctrl-C keeps what ran): its record with schema, bench (when the bench started, which groups its runs), version, scenario and run. troupe bench --compare puts the last bench beside each scenario's last runs in an earlier one; --compare <version|model> beside that version's or model's last runs. The file is the person's, beside their sessions, and holds what runs did and cost, never what they were sent. - --repeat N, default 1. One run of each is the cheapest answer to "does it work here and what does it cost"; to compare, three or more. The issue's open question (3 or 5) is the nightly's. - The nightly, later. It never runs for a pull request.

Governs: apps/troupe_core/lib/troupe/bench.ex, apps/troupe_core/lib/troupe/bench/history.ex, apps/troupe_core/lib/troupe/bench/live.ex, apps/troupe_core/lib/troupe/bench/live_scenarios.ex, apps/troupe_core/lib/troupe/bench/scenario.ex, apps/troupe_core/lib/troupe/llm/provider.ex, apps/troupe_core/lib/troupe/reaper.ex, clients/tui/lib/troupe/cli/bench.ex