The inference paradox

Two Gartner forecasts from this year look contradictory until you read them together.

In March, Gartner predicted that by 2030, running inference on a 1-trillion-parameter LLM will cost providers over 90% less than it did in 2025. The same press release added two caveats: providers won't pass all of the savings on to customers, and token consumption will grow faster than costs fall.

In August, Gartner forecast that inference costs per agentic workflow will rise more than fivefold through 2028. VentureBeat's coverage explains the mechanics. Sending a task to an agentic reasoning model costs providers at least five times as much as a basic chatbot exchange, and agents use roughly 5 to 30 times more tokens for the same task.

Both forecasts can be right at once. The price of a token falls. The number of tokens per job rises faster. The bill goes up.

Spending is rising with it. According to CIO.com's report on Gartner's September forecast, worldwide AI spending will reach $2.7 trillion in 2026, up 49.5%, with spending on AI agents and assistants up 77.3%. A lot of companies are about to discover their agent costs at the same time.

Where agent budgets actually go: refinement loops

The most useful figure we've seen this year on agent economics comes from McKinsey. A July 2026 McKinsey Quarterly article, "Is that AI agent worth it?", reports that about 60% of agentic AI spend goes to response refinement: the loops where an agent checks its own output and redoes it. A McKinsey FinOps survey from May found 93% of respondents over their AI budgets, though that sample was small, at 75 respondents. Both figures reach us through MarketScale's coverage, not the original reports, so treat them as indicative.

The 60% figure won't surprise anyone who has read a long agent trace. The first draft is cheap. What costs money is the agent that reviews the draft, decides it isn't good enough, re-reads the whole context and tries again, often on the same expensive model as the first pass because nobody configured it otherwise.

Refinement isn't waste in itself. Checking work is how agents reach acceptable accuracy. The waste is refining everything, at frontier prices, with no cap on iterations.

Tiering and routing: put the right model on each step

Gartner's recommendation is inference tiering and routing, and current price lists make the case on their own. According to the Claude docs, Claude Haiku 5.5 input starts at $0.10 per million tokens, against $4 for Opus 5.5. That's a 40x gap inside a single vendor's lineup.

A worked example at list prices. A classification step that reads 3,000 tokens and writes 200 costs about $0.016 on Opus 5.5 ($0.012 in, $0.004 out) and about $0.0004 on Haiku 5.5. Run that step 100,000 times a day and it's roughly $1,600 a day against $40, for one step. If Haiku classifies as well as Opus on your data (and only your eval set can tell you that), routing that single step pays for a lot of engineering time.

Most agent pipelines have a few steps that need a frontier model, usually planning, the hard reasoning step and final synthesis, and many that don't: classifying the input, extracting fields, choosing a tool, checking a format. Make the routing decision per step, with evidence, and revisit it whenever prices change.

Zenyt is built along these lines. Its agents audit e-commerce stores the way shoppers browse them (product information, visual content, functionality, checkout, SEO, broken links), with multi-provider LLM routing underneath. The same architecture runs unchanged from 50-product boutiques to million-page enterprise catalogs. At a million pages, per-page cost decides whether the product works commercially at all.

Cache, batch, verify selectively: patterns from production

After routing, three habits do most of the work.

Cache what repeats. Per the Claude docs, prompt-cache reads are billed at a fraction of the base input price (10% on Haiku 5.5). Agents resend the same system prompt, tool definitions and reference documents on every turn. Put the stable parts first, keep them byte-identical between calls, and treat the cache hit rate as a metric someone watches. A template change that moves a timestamp to the top of the prompt can quietly wipe out your caching, and nothing will fail except the invoice.

Batch what can wait. Batch processing is 50% off, per the same docs. Nightly re-audits, bulk document extraction, backfills and evaluation runs rarely need an answer in seconds. If a job can tolerate a delay of hours, it shouldn't pay real-time prices.

Verify selectively. This is where the refinement budget is won or lost. On Zenyt, detection runs in two stages: a first pass flags potential issues, and a second stage re-investigates each one with screenshot proof before anything is reported. The expensive verification runs on candidates, not on every page, and that reflection loop is also what dramatically reduced false positives.

The general rule we apply: refinement should be triggered by a signal (a low confidence score, a failed validation, a flagged anomaly) and it should be capped. Two retries, then escalate to a human or a stronger model. An open-ended "keep improving until it's good" loop is how one strange input eats a day's budget.

Measuring cost per outcome, not cost per token

Cost per token is the vendor's metric. Cost per outcome is yours: per resolved ticket, per audited page, per answered research question, per processed document.

TaxFigure shows why the difference matters. A research coordinator splits each tax question into parallel sub-tasks, specialized agents search statutes, regulations, case law and client documents simultaneously, and a synthesis agent writes the answer with full citations. A per-token dashboard would tell you which model is expensive. It wouldn't tell you what an answered question costs, or whether a cheaper retrieval model increased the number of answers that needed rework.

Promptify makes the same point from another direction. Its execution engine sits in front of several model providers, so each prompt template can run on the model that fits its cost and quality needs. In systems shaped like that, cost is driven by how many steps run and how much context each step carries, more than by the price of any single call.

So tag every model call with the workflow, the step, the customer and an outcome ID, and roll the numbers up. In our AI deployment program, the production handoff includes a monitoring dashboard so the client's team can see what each agent is doing. Cost per outcome belongs on that screen, next to accuracy, where the people who own the workflow will actually see it.

A FinOps checklist for agentic systems

  • Tag every call with workflow, step, tenant and outcome, and report cost per outcome every week.
  • Assign a model tier to each step based on eval results, and review the assignments when prices change.
  • Cap refinement loops, and trigger them on signals rather than by default.
  • Track cache hit rate per prompt template, with an alert when it drops.
  • Move anything that can wait hours to batch.
  • Set per-workflow budgets with alerts at 50% and 80%, plus a hard stop for runaway loops.
  • Re-run the eval harness before switching models, and again a week after.

Cheaper tokens will keep arriving, and we think they will change less than people expect. A workflow that burns ten times the tokens it needs stays expensive at any price per token, so the teams that come out ahead will be the ones that can already name the cost of one resolved ticket. If your bill is growing faster than your usage, we can help you find out why.

The question we keep coming back to: once vendors can see which agent runs produce outcomes, how long before they stop pricing per token and start pricing per outcome?

Sources