The Evaluation Cost Ratio

Agentic Systems

India Market

Product Economics

LLM-as-judge pricing works in US markets where ARPU is $500/month but breaks Indian edtech at ₹200/month.

Author

B. Talvinder

Published

March 5, 2026

Galileo raised $45 million in October 2024 to build AI evaluation tools. Revenue grew 834% that year. Six Fortune 50 companies signed up. Snorkel AI raised $100 million in May 2025 at a $1.3 billion valuation, with Snorkel Evaluate as a core product. Confident AI came out of YC W25.

These companies are now pitching Indian edtech buyers. Beautiful decks. Impressive demos. Per-evaluation API pricing that will bankrupt every Indian buyer who signs.

I keep watching this happen. The demo works. The pricing model is imported from a market where course fees are $500-2,000/seat. The Indian buyer is selling at ₹200-800/learner/month. Nobody does the math until after the contract is signed.

The number that kills you

Take a corporate training product. 10,000 active learners, 4 evaluations per month. 40,000 evaluation events.

LLM-as-judge at GPT-4o-mini rates — $0.15 per million input tokens, $0.60 per million output tokens. A basic evaluation prompt with rubric and response runs roughly 1,500 tokens in, 500 out. That’s about $0.0005 per evaluation. At 40,000 evaluations: $20/month. Sounds fine.

Now make the evaluation useful. Detailed feedback, multi-criteria scoring, follow-up questions. You’re using 4,000 tokens in, 2,000 out. Per-eval cost jumps to $0.0018. Still small. But you want GPT-4o quality for nuanced judgment — $2.50/$10 per million tokens. Now you’re at $0.03 per evaluation. $1,200/month. $1.20 per learner per month.

Your Indian enterprise client is paying ₹200/learner/month. That’s roughly $2.40.

Evaluation Cost Ratio = evaluation spend / per-learner revenue.

At $0.03/eval with GPT-4o: 50%. Half your revenue on evaluation alone.

Market	Monthly ARPU	Eval cost/learner/month
US enterprise training	$500	$1.20
US mid-market	$50	$1.20
Indian corporate training	$2.40	$1.20

The evaluation startups aren’t lying. Their product works. It works in markets where the ECR is under 3%. In Indian markets, the same product eats half your revenue. The engineering is impressive. The product-market fit is nonexistent.

India already solved this problem

JEE Main evaluates 1.3 million candidates annually. GATE 2025 had 7.37 lakh candidates appear across all papers. No LLM. No army of graders. Structured assessment at Indian scale, Indian price points.

The GMAT removed its Analytical Writing Assessment entirely when it launched the Focus Edition in 2023 — made the exam an hour shorter by cutting the essay. Then brought it back in July 2024 as an optional “Business Writing Assessment” after business schools complained they couldn’t tell if applicants or ChatGPT wrote the essays. The lesson: subjective evaluation keeps getting harder and more expensive. Structured evaluation keeps scaling.

Physics Wallah scaled to 4.46 million paid users in FY25, up from 1.76 million in FY23. Revenue crossed ₹3,000 crore. Online ACPU was ₹3,682. Their bottleneck was never evaluation — it was content production and offline expansion. They solved the right scaling problem.

The insight isn’t new. Structure the assessment so it’s objective AND scalable. India cracked this decades ago for science and math. The product opportunity is applying the same principle to judgment skills — leadership decisions, case analyses, strategic thinking — that the exam tradition doesn’t handle well.

What I actually built

I ran into this wall building Ostronaut’s training platform. We generate learning content with AI — slides, games, interactive scenarios. The generation pipeline uses LLMs heavily. Expensive per content piece, but it’s a one-time cost amortized across all learners who consume it.

The evaluation architecture is completely different. For game-based scenarios — card games and turn-based simulations that teach decision-making — scoring is rule-based. The system defines optimal play. Scores against it. Runs in milliseconds. Costs nothing at the margin. Same input, same score, every time.

LLM creates the scenario. Rules judge every move within it.

I use LLM judgment in exactly one place: validating generated content before it reaches learners. One validation pass per content piece. That cost scales with production volume, not learner volume. The difference matters.

	Content creation	Learner evaluation
Method	LLM generation + LLM validation	Rule-based scoring
Cost structure	One-time per piece	Per-interaction
Scales with	Content volume (manageable)	Learner volume (must approach zero)

Get this split wrong and you bleed money from day one.

Manufacturing figured this out fifty years ago

You can inspect every widget coming off the line, or you can design the production process so widgets come out right. Inspection scales linearly with output. Built-in quality is expensive upfront and free at scale.

LLM-as-judge is inspection. Structured rubrics with rule-based scoring is built-in quality.

I watch smart founders import the inspection model from Western markets, build their entire evaluation architecture around it, and then discover nine months later that their unit economics don’t work. By then they’ve raised on metrics that assumed evaluation costs would decrease. They won’t. They scale linearly with learner volume.

What I’m not sure about

The ECR math is clear at current model prices. But model prices are dropping fast. GPT-4o-mini already costs 60% less than GPT-3.5 Turbo did at launch. If evaluation costs fall another 10x in two years, does the ECR problem solve itself?

Maybe. If GPT-5-equivalent evaluation costs $0.003/eval, the Indian ECR drops to about 5%. Livable. But I’ve watched this movie before with cloud storage, with compute, with bandwidth. Prices drop, but usage grows faster. You build assuming the cost decrease, then discover you’re evaluating 10x more often because you can. The ECR stays broken.

The other question: are there domains where LLM-as-judge is the only option? Creative writing feedback, strategic case analysis, nuanced communication skills — these don’t reduce cleanly to rules. Maybe the answer is tiered: rule-based evaluation for 80% of learners, LLM evaluation for the premium 20% who pay 5x more. I haven’t seen anyone execute this successfully yet.

The pattern I keep seeing is founders who treat evaluation as a feature, not a cost center. They assume it’ll be cheap because the demo was cheap. Then they scale to 50,000 learners and the AWS bill is suddenly larger than payroll.

The evaluation startups are solving a real problem. Just not for Indian markets. Not at these price points. Not yet.

--- title: "The Evaluation Cost Ratio" description: "LLM-as-judge pricing works in US markets where ARPU is $500/month but breaks Indian edtech at ₹200/month." date: 2026-03-05 categories: [Agentic Systems, India Market, Product Economics] draft: false --- Galileo raised $45 million in October 2024 to build AI evaluation tools. Revenue grew 834% that year. Six Fortune 50 companies signed up. Snorkel AI raised $100 million in May 2025 at a $1.3 billion valuation, with Snorkel Evaluate as a core product. Confident AI came out of YC W25. These companies are now pitching Indian edtech buyers. Beautiful decks. Impressive demos. Per-evaluation API pricing that will bankrupt every Indian buyer who signs. I keep watching this happen. The demo works. The pricing model is imported from a market where course fees are $500-2,000/seat. The Indian buyer is selling at ₹200-800/learner/month. Nobody does the math until after the contract is signed. ## The number that kills you Take a corporate training product. 10,000 active learners, 4 evaluations per month. 40,000 evaluation events. LLM-as-judge at GPT-4o-mini rates — $0.15 per million input tokens, $0.60 per million output tokens. A basic evaluation prompt with rubric and response runs roughly 1,500 tokens in, 500 out. That's about $0.0005 per evaluation. At 40,000 evaluations: $20/month. Sounds fine. Now make the evaluation useful. Detailed feedback, multi-criteria scoring, follow-up questions. You're using 4,000 tokens in, 2,000 out. Per-eval cost jumps to $0.0018. Still small. But you want GPT-4o quality for nuanced judgment — $2.50/$10 per million tokens. Now you're at $0.03 per evaluation. $1,200/month. $1.20 per learner per month. Your Indian enterprise client is paying ₹200/learner/month. That's roughly $2.40. **Evaluation Cost Ratio = evaluation spend / per-learner revenue.** At $0.03/eval with GPT-4o: 50%. Half your revenue on evaluation alone. | Market | Monthly ARPU | Eval cost/learner/month | ECR | |---|---|---| | US enterprise training | $500 | $1.20 | 0.24% | | US mid-market | $50 | $1.20 | 2.4% | | Indian corporate training | $2.40 | $1.20 | 50% | The evaluation startups aren't lying. Their product works. It works in markets where the ECR is under 3%. In Indian markets, the same product eats half your revenue. The engineering is impressive. The product-market fit is nonexistent. ## India already solved this problem JEE Main evaluates 1.3 million candidates annually. GATE 2025 had 7.37 lakh candidates appear across all papers. No LLM. No army of graders. Structured assessment at Indian scale, Indian price points. The GMAT removed its Analytical Writing Assessment entirely when it launched the Focus Edition in 2023 — made the exam an hour shorter by cutting the essay. Then brought it back in July 2024 as an optional "Business Writing Assessment" after business schools complained they couldn't tell if applicants or ChatGPT wrote the essays. The lesson: subjective evaluation keeps getting harder and more expensive. Structured evaluation keeps scaling. Physics Wallah scaled to 4.46 million paid users in FY25, up from 1.76 million in FY23. Revenue crossed ₹3,000 crore. Online ACPU was ₹3,682. Their bottleneck was never evaluation — it was content production and offline expansion. They solved the right scaling problem. The insight isn't new. Structure the assessment so it's objective AND scalable. India cracked this decades ago for science and math. The product opportunity is applying the same principle to judgment skills — leadership decisions, case analyses, strategic thinking — that the exam tradition doesn't handle well. ## What I actually built I ran into this wall building Ostronaut's training platform. We generate learning content with AI — slides, games, interactive scenarios. The generation pipeline uses LLMs heavily. Expensive per content piece, but it's a one-time cost amortized across all learners who consume it. The evaluation architecture is completely different. For game-based scenarios — card games and turn-based simulations that teach decision-making — scoring is rule-based. The system defines optimal play. Scores against it. Runs in milliseconds. Costs nothing at the margin. Same input, same score, every time. LLM creates the scenario. Rules judge every move within it. I use LLM judgment in exactly one place: validating generated content before it reaches learners. One validation pass per content piece. That cost scales with production volume, not learner volume. The difference matters. | | Content creation | Learner evaluation | |---|---|---| | Method | LLM generation + LLM validation | Rule-based scoring | | Cost structure | One-time per piece | Per-interaction | | Scales with | Content volume (manageable) | Learner volume (must approach zero) | Get this split wrong and you bleed money from day one. ## Manufacturing figured this out fifty years ago You can inspect every widget coming off the line, or you can design the production process so widgets come out right. Inspection scales linearly with output. Built-in quality is expensive upfront and free at scale. LLM-as-judge is inspection. Structured rubrics with rule-based scoring is built-in quality. I watch smart founders import the inspection model from Western markets, build their entire evaluation architecture around it, and then discover nine months later that their unit economics don't work. By then they've raised on metrics that assumed evaluation costs would decrease. They won't. They scale linearly with learner volume. ## What I'm not sure about The ECR math is clear at current model prices. But model prices are dropping fast. GPT-4o-mini already costs 60% less than GPT-3.5 Turbo did at launch. If evaluation costs fall another 10x in two years, does the ECR problem solve itself? Maybe. If GPT-5-equivalent evaluation costs $0.003/eval, the Indian ECR drops to about 5%. Livable. But I've watched this movie before with cloud storage, with compute, with bandwidth. Prices drop, but usage grows faster. You build assuming the cost decrease, then discover you're evaluating 10x more often because you can. The ECR stays broken. The other question: are there domains where LLM-as-judge is the only option? Creative writing feedback, strategic case analysis, nuanced communication skills — these don't reduce cleanly to rules. Maybe the answer is tiered: rule-based evaluation for 80% of learners, LLM evaluation for the premium 20% who pay 5x more. I haven't seen anyone execute this successfully yet. The pattern I keep seeing is founders who treat evaluation as a feature, not a cost center. They assume it'll be cheap because the demo was cheap. Then they scale to 50,000 learners and the AWS bill is suddenly larger than payroll. The evaluation startups are solving a real problem. Just not for Indian markets. Not at these price points. Not yet.