Measuring AI ROI: 4 Metrics That Matter (And 3 Vanity Metrics to Ignore)

Pankaj Kumar Pandey

VP-Engineering

I've spent more than a decade building software and leading engineering teams at startups and growing companies. These days I help founders and engineering managers work out where AI will pay for itself, and where it won't. I write here about the decisions that come before the code.

"We processed 40 million tokens this month" tells your board how much you spent, not what you got. AI earns its place only when a business number moves.

In IBM's 2025 survey of 2,000 CEOs, they reported that only a quarter of their AI initiatives had delivered the expected return (IBM, 2025):

Share of AI initiatives, as reported by CEOs

Expected ROI 25%

Scaled widely 16%

Most initiatives can't show a return. Often nobody measured the "before".

Indian companies are moving fast: 47% now have several GenAI use cases live (EY, 2025). The question is which numbers prove they're worth keeping.

Four metrics that matter

Cycle time. How long one unit of work takes, from request to done. A claim, an invoice, a support ticket.
Error rate. The share of outputs that need rework, refunds or apologies.
Cost per transaction. People time plus API bill plus tools, divided by volume.
Headcount pressure. Hiring you no longer need as volume grows. Saved hiring, not layoffs.

Three vanity metrics to ignore

Tokens or calls. That's your bill, not your benefit.
Model size or benchmark scores. A bigger model on the wrong task is still the wrong task.
Number of pilots. Ten pilots and no production use is ten costs.

An illustrative before-and-after (example figures)
Metric	Before	After
Cycle time per invoice	2 days	4 hours
Error rate	6%	2%
Cost per invoice	₹120	₹45
Hires needed next year	4	1

Check your measurement

Score each 0 (no), 1 (partly) or 2 (yes):

Baseline: all four metrics were measured before launch.
Owner: a business owner, not the AI team, reports them.
Full cost: people time and API bills are both counted.
Cadence: the numbers are reviewed monthly.
Stop rule: there's a date to kill it if nothing moves.

What your score means (my recommendation)
Score	Next move
8–10	Report it to the board as a business result.
5–7	Measure a baseline on one untouched process now.
0–4	Pause expansion until you can prove one number moved.

Your next step

For your biggest AI initiative, write the four numbers in two columns: before and now. If you can't fill the "before" column, measure the same process where AI isn't used yet, this week.

Sources

IBM (2025). IBM Study: CEOs Double Down on AI While Navigating Enterprise Hurdles. 6 May 2025.
EY (2025). The AIdea of India: Outlook 2026. EY and CII, 16 November 2025.

NEW CODE EVERY MONTH.

Get a note when new Claude Code templates, code recipes or tools go live.

No account required

We email once to confirm. Unsubscribe anytime — it deletes your address.