Skip to content

Running the throughput test

The bench dials a Troupe worker directly, with tokens it mints itself. That works because a worker verifies session tokens offline against a JWKS on disk; hand it a JWKS you hold the private half of and it will accept your tokens exactly as it would the plane's. This is for measuring, never for a deployment.

A worker with no plane, no object store and the scripted fake model measures a client: the path from a frame to the session actor and back, without a model's latency. It runs as one Docker container, dialled with tokens the bench mints itself (packages/bench/src/devToken.ts). The sections below are the key pair, the docker run, the run and its variables.

Each client opens its own socket and its own session (five clients on one session would measure the model's serialisation, not the transport) and sends its prompts one at a time. Every prompt is timed from just before input.send to the ack, input_accepted, the first llm_delta, the durable llm_response and the end of the turn. Results are printed and written as JSON under packages/bench/bench-results/.

1. A key pair and a JWKS

mkdir -p keys && cd keys
node -e '
const { generateKeyPairSync, createHash } = require("crypto");
const fs = require("fs");
const { privateKey, publicKey } = generateKeyPairSync("ec", { namedCurve: "P-256" });
const pub = publicKey.export({ format: "jwk" });
const priv = privateKey.export({ format: "jwk" });
// RFC 7638 thumbprint: the plane and the worker both compute `kid` this way.
const canonical = JSON.stringify({ crv: pub.crv, kty: pub.kty, x: pub.x, y: pub.y });
const kid = createHash("sha256").update(canonical).digest("base64url");
fs.writeFileSync("jwks.json", JSON.stringify({ keys: [{ ...pub, kid, use: "sig", alg: "ES256" }] }));
fs.writeFileSync("signing-key.json", JSON.stringify({ ...priv, kid }));
'
cat > fake.json <<'JSON'
{"steps": [{"text": "Hello from the fake model. A few sentences, so the client sees a stream of deltas rather than one word."}]}
JSON
cd ..

2. A worker

docker run -d --name troupe-bench-worker -p 4000:4000 \
  -v "$PWD/keys:/etc/troupe:ro" \
  --tmpfs /workspace:uid=1000,gid=1000,size=512m \
  -e TROUPE_WORKER_AUTOSTART=true -e TROUPE_PROFILE=bench -e TROUPE_POD_ORDINAL=bench-0 \
  -e TROUPE_JWKS_PATH=/etc/troupe/jwks.json \
  -e TROUPE_PROVIDER=fake -e TROUPE_MODEL=fake -e TROUPE_FAKE_SCRIPT=/etc/troupe/fake.json \
  -e TROUPE_STATE_HOME=/workspace/.state -e TROUPE_SESSIONS_PER_POD=64 \
  -e TROUPE_HTTP_PORT=4000 -e RELEASE_DISTRIBUTION=none -e ERL_FLAGS="+Q 65536" \
  ghcr.io/it-minds/troupe-worker:dev    # built here by scripts/build-images troupe_worker; CI publishes no :dev

# one workspace per client; the worker will not invent directories
docker exec troupe-bench-worker sh -c 'for i in $(seq 0 63); do mkdir -p /workspace/c$i; done'

TROUPE_POD_ORDINAL is the worker id and therefore the token audience; a token minted for any other aud is refused with wrong_audience. ERL_FLAGS=+Q 65536 matters: the BEAM sizes its port table from RLIMIT_NOFILE, which Docker sets absurdly high.

On Git Bash for Windows, prefix docker commands with MSYS_NO_PATHCONV=1 so /etc/troupe and /workspace are not rewritten into C:/Program Files/Git/....

3. The run

BENCH_SIGNING_KEY="$PWD/keys/signing-key.json" BENCH_POD_ID=bench-0 \
BENCH_CLIENTS=20 BENCH_PROMPTS=10 pnpm bench

pnpm bench runs in packages/bench, so a relative path in a variable is read from there.

variable default meaning
BENCH_MODE worker worker dials BENCH_WS with self-minted tokens; plane goes through BENCH_PLANE with BENCH_PLANE_TOKEN and BENCH_PROFILE
BENCH_WS ws://localhost:4000/v1/socket the worker
BENCH_SIGNING_KEY none; worker mode needs it the private half of the JWKS the worker trusts, which the bench signs its tokens with
BENCH_POD_ID none; worker mode needs it the worker's id, TROUPE_POD_ORDINAL: the tokens' audience
BENCH_CLIENTS 5 concurrent clients, one session each
BENCH_PROMPTS 20 prompts per client, one in flight at a time
BENCH_WORKSPACE /workspace per-client sessions are created under <workspace>/c<n>
BENCH_OUT bench-results where the JSON goes, under packages/bench when relative

Keep prompts per client under 40, or raise the session's max_turns: the default turn budget ends a session in a way that looks exactly like a throughput problem.

packages/bench/src/trace.ts connects once, sends one prompt and prints every frame; use it when something in the run above does not add up.

4. What it measured

Measured on 2026-09-12, Docker Desktop on a Windows laptop, worker image built from commit 2f85e39 of the platform's former repository, fake model:

clients × prompts connect+init p50 input_accepted p50 / p95 llm_response p50 / p95 turn end p50 / p95 turns/s
5 × 20 13 ms 0.6 / 1.3 ms 1.1 / 3.7 ms 1.3 / 9.1 ms ~900
20 × 10 44 ms 0.9 / 2.1 ms 3.7 / 5.6 ms 4.1 / 7.2 ms ~1000

All 300 turns completed; the fake model answers instantly, so these numbers are the harness and the transport, not a model. BENCH_MODE=plane runs the same test through a plane; that path has not been measured, since it needs Postgres, OpenBao and an identity provider behind it.