.png)
Cheaper tokens are not making enterprise AI cheaper.
The industry is beginning to see it. Goldman Sachs Research expects agentic AI to drive a 24× increase in token consumption by 2030. EY says model routing has reduced its token use by up to 60%. Gartner is urging leaders to manage cost per result — not simply cost per request.
The realization is spreading. But there is still a gap between recognizing the problem and designing the system around it.
Most advice starts at the API layer: cache more, batch more, switch to a cheaper model, use open source. All useful but usually sufficient. Because each of those levers is a domain question wearing a cost costume. Route to the right model — right for which task, verified how? Trim your context — to what? The generic advice stops precisely where the engineering begins.
At Menos, building AI research intelligence for institutional asset managers, we've spent our time on the part after the advice ends.
Three things we've learned:
1. You can't route what you can't measure.
EY's result validates the lever that routing works. But routing is only as good as the evaluation behind it, and generic benchmarks can't tell you which model clears the quality bar for your tasks.
We built our own: a suite of production tasks — including but not limited to structured trade idea extraction, research planning, multi-source synthesis, financial computation, narrative graph decomposition — with 1,000+ cases across asset classes and tasks.
The design decision that mattered most: metrics that encode domain semantics rather than string comparison. Example — a sell-side note says "SGD OIS 1x5 flattener." The system must decompose this into two legs (Pay SGD 1Y OIS, Receive SGD 5Y OIS), classify it as a single curve trade, and tag multi-leg type accordingly. No public benchmark tests this. Our custom eval does, because this is the actual work the model has to do — and a systematic error here means retries, analyst corrections, and tokens spent twice.
That evaluation system re-tests every model release and re-tunes routing as capabilities and prices shift. Routing isn't a config you set once. It's a continuous measurement discipline — and the measurement itself requires domain expertise.
2. The cheapest token is the one you never send.
"Trim your context" assumes you already know what's relevant. That's a domain question, not a compression trick.
→ Minimal context. Because the knowledge layer we built for the clients has already decomposed each research note into typed structures — narratives with direction and conviction, trade ideas with tenor and strikes, causal graphs linking evidence to views — a query pulls the precise structured slice it needs, not 40 raw PDFs in context.
→ Bounded outputs. Output tokens cost 4–5× input tokens across every major provider. Schema-constrained generation keeps responses short and targeted — same answer, a fraction of the output bill.
→ Zero-token validation. Not every check needs a model. A deterministic validator catches fabricated citation keys without a single LLM call — cheaper and more reliable than asking a judge model to verify every claim.
→ Batch where latency doesn't matter. Ingestion-time classification has no user waiting on it — run it in batch at roughly half the real-time cost.
3. Process once, amortize forever.
This is where the economics invert — and where the knowledge layer does double duty.
Every document is classified, extracted, and structured at ingestion. But "structured" doesn't mean "pre-processed and shelved." It means every research note feeds a living knowledge architecture: which market narratives are active, where analyst theses conflict, how conviction on a view has shifted over six months. The expensive cognition happens once; every analyst and every downstream query draws on the compounding result.
Structured context is also stable context — it caches at up to 90% off the repeated portion. And as adoption grows, the per-query cost falls instead of climbing. That's not a line item someone negotiated. It's the architecture working as designed.
The gap the industry is closing is real, and the progress is worth celebrating. But the deeper truth is this: token efficiency isn't a FinOps function you bolt onto your AI stack after the fact. It's a property of how well your system understands your domain — because the same knowledge that makes the answer right the first time is what tells you which tokens you never needed to spend.
