Chain-of-Thought Has an Efficiency Tax
Your AI agent now “thinks through” problems step-by-step. Your token costs just tripled. Did anyone on your team notice?
Chain-of-thought prompting is the default recommendation for improving LLM output quality. Every tutorial says it. Every framework enables it. And the advice is correct — CoT does improve reasoning on complex tasks. What nobody mentions is the cost.
Every major model provider now ships a reasoning mode — extended thinking, chain-of-thought, “deep research.” These modes generate 3x to 10x more tokens than their standard equivalents for the same task. Those tokens cost money. They add latency. And in most production systems, nobody is measuring whether the quality improvement justifies the spend.
The numbers
Here’s what the efficiency tax looks like in practice.
A content generation agent running a standard frontier model averages 1,200 input tokens and 800 output tokens per query. That’s roughly $0.036 per call at current pricing. Switch to the same provider’s reasoning mode, and the same task burns 1,200 input tokens plus 4,000-8,000 thinking tokens plus 1,200 output tokens. Cost per call: $0.12 to $0.22. A 3x to 6x increase.
At 10 queries a day, nobody cares. At 10,000 queries a day, you’ve added $840 to $1,840 in daily costs for a quality improvement you probably haven’t measured.
| Approach | Tokens per call | Cost per call | Monthly cost at 10K/day |
|---|---|---|---|
| Direct prompting (standard mode) | ~2,000 | ~$0.036 | ~$10,800 |
| Chain-of-thought (standard mode) | ~3,500 | ~$0.055 | ~$16,500 |
| Reasoning mode (extended thinking) | ~8,000 | ~$0.18 | ~$54,000 |
That last row is 5x the first row. For many tasks, the output quality difference between row one and row three is negligible.
Why teams don’t measure this
Three reasons, all predictable.
Accuracy bias. Teams optimizing AI systems measure quality metrics — accuracy, coherence, task completion rate. Token efficiency rarely appears on the dashboard. When the reasoning model produces a slightly better answer, that’s visible. The 5x cost increase lives in a billing page nobody checks until month-end.
Scale hiding the problem. At low volumes, the tax is invisible. A startup running 500 queries a day doesn’t feel the difference between $18 and $90. But costs scale linearly with volume, and the moment you hit product-market fit, the tax becomes your second-largest line item after engineering salaries.
The “just optimize later” fallacy. This is the same mistake teams make with database queries. Ship first, optimize later. Except “later” usually means “after we’ve built the entire pipeline around the expensive approach and switching costs are enormous.”
When the tax is worth paying
CoT and reasoning models earn their cost in specific situations.
Multi-step logical reasoning. Tax-return calculations, legal document analysis, complex debugging. Tasks where the intermediate steps actually matter for correctness. The tax is justified because direct prompting fails outright.
Low-volume, high-stakes decisions. Medical triage recommendations, financial risk assessments, safety-critical systems. When a single wrong answer costs more than a thousand correct ones, pay the tax.
Tasks where you can measure the delta. If you can run both approaches on the same inputs and quantify the accuracy improvement, you can calculate the break-even. Most teams skip this step.
Where the tax is almost never worth it: classification tasks, structured data extraction, template-based generation, summarization, entity recognition. These tasks work well with direct prompting. CoT adds cost without proportional benefit.
The metric that matters
Cost per unit of quality. Not just cost per query, and not just quality per query — the ratio.
Define a quality metric for each task type. Run both approaches on a held-out sample. Calculate cost-per-quality-point for each. If reasoning-mode costs 5x more but only improves quality by 8%, that’s a bad trade. If it costs 3x more and improves quality by 40%, that might be worth it.
This is basic FinOps thinking applied to LLM inference. Cloud cost optimization is a mature discipline — rightsizing instances, reserved capacity, spot pricing. LLM cost optimization is in its infancy. Most teams are running the equivalent of on-demand instances at maximum size for every workload.
A practical approach
Route by task complexity. Not every request needs to go through the reasoning model. Build a classifier — a cheap, fast one — that scores incoming tasks on complexity. Simple tasks go to the direct model. Complex tasks go to the reasoning model. This is the same pattern as CDN edge routing: serve what you can cheaply at the edge, send the rest to origin.
At Ostronaut, we found that roughly 70% of content generation tasks hit a template fast path — no reasoning needed. The remaining 30% benefit from deeper processing. Routing saves more than optimizing any single model call.
The irony is that the routing classifier itself is a trivial LLM call. A $0.001 classification that saves $0.15 on the main call pays for itself in a single interaction.
What I don’t track yet
Latency cost. The efficiency tax isn’t just financial — reasoning models are slower. Time-to-first-token increases. Total response time increases. For interactive applications, the user experience degradation has a cost that doesn’t show up on any invoice.
I don’t have a clean way to quantify the latency tax in dollar terms. The financial tax is measurable today. The latency tax needs better tooling. If you’re building internal AI platforms, this is the metric to add next.
The teams that measure both — cost efficiency and latency efficiency, per task type — will have a significant operational advantage over the teams that just pick the most powerful model and hope for the best.