Skip to content

Token usage is four disjoint figures — input_tokens, cache_read, cache_write, output_tokens — the budget charges what was billed, and compaction and the context gauge measure the prompt's whole length

Decision 657 · accepted · 2026-09-20

Anthropic's input_tokens excludes what its prompt cache served and OpenAI's prompt_tokens includes it, so read raw the same conversation counts differently depending on who answered, and each is wrong in the direction that hurts. On an OpenAI-compatible provider a long conversation re-reads its whole prompt every turn and nearly all of it is a cache read billed at a tenth, so max_input_tokens would exhaust roughly ten times early, on work the user was barely paying for; on Anthropic a warm 200k conversation reads as a few hundred input tokens once the cache hits, so compaction would never fire. Each adapter converts at the boundary — OpenAI's prompt_tokens_details.cached_tokens comes back out of prompt_tokens; Anthropic's cache_read_input_tokens and cache_creation_input_tokens are read beside input_tokens, and every figure a message_delta reports replaces the running total — so input_tokens + cache_read + cache_write is the prompt's length whoever answered. Budget.charge_usage/2 spends Usage.billed_input/1 (fresh input plus cache writes), and the agent's last_input_tokens, which needs_compaction? and Headroom's context read, is Usage.total_input/1. The llm_response event carries all four keys, and Usage.from_json/1 folds an event written before them as a prompt nothing was cached of, which is what it was. Troupe.Session.Usage, the ledger, reads input_tokens, the uncached figure, because the cost it records comes from the gateway and not from the tokens; the cache figures ride in the event for the day the ledger wants them.

Governs: apps/troupe_core/lib/troupe/agent/headroom.ex, apps/troupe_core/lib/troupe/budget.ex, apps/troupe_core/lib/troupe/llm/message.ex, apps/troupe_core/lib/troupe/llm/providers/anthropic.ex, apps/troupe_core/lib/troupe/llm/providers/openai.ex