The AI budget conversation usually starts in engineering and ends in finance, and the two sides are often speaking different languages. Engineers talk about token limits, inference latency, and model tiers. Finance teams want to know why the quarterly compute bill tripled and what the plan is to bring it under control.
That disconnect is expensive, not just in the obvious sense – runaway API costs and over-provisioned GPU clusters -, but in the strategic sense. Organizations that don’t build cost discipline into their AI operations from the start tend to hit a ceiling: the unit economics of their AI initiatives don’t hold up at scale, and what looked like a strong business case in the pilot falls apart in production.
According to ICONIQ’s 2026 State of AI report, internal AI spend at software companies building AI products has climbed from 11% of revenue in 2025 to a projected 16% in 2026, on track to reach 19% by 2027.1 For enterprises adding AI capabilities to existing products and workflows, the picture is less extreme but the trend is the same: a 2026 Statista survey of 2,360 organizations found the average AI investment share of revenue across industries more than doubling, from 0.8% in 2025 to a projected 1.7% in 2026.2
This article breaks down the five main cost drivers in enterprise AI operations, the waste patterns that make each one worse than it needs to be, and the control levers your team can apply without sacrificing capability.
Where the money actually goes
Enterprise AI costs cluster around five primary drivers: token usage, model tier selection, inference compute, retry and failure overhead, and embedding and vector storage. Most teams have visibility into one or two of these. Few have a consolidated view across all five, which is where the surprises tend to come from.
Cost driver
What drives it
Common waste pattern
Control lever
The table above maps each driver to its most common waste pattern and the control lever that addresses it. The rest of this article works through each lever in detail.
Token costs: the bill that scales with every prompt
Token-based pricing is the most visible AI cost, and the one most susceptible to waste that accumulates quietly at scale. Every API call to a hosted model is billed by the number of input tokens (your prompt, context, conversation history) plus output tokens (the model’s response). The math is simple; the discipline to manage it is less common.
The most prevalent waste pattern is verbose system prompts sent on every call. A system prompt that runs to 2,000 tokens, delivered with each of 10,000 daily requests, adds up to 20 million tokens per day in prompt overhead alone, before the actual user query lands. Auditing and compressing system prompts is the fastest cost reduction lever most teams have available, and it requires no model changes.
Prompt caching addresses the same problem at the infrastructure level. Most major model providers now offer caching for repeated prompt prefixes; if your system prompt is identical across calls, you pay a fraction of the cost for cached tokens versus fresh ones. Anthropic reports that caching can cut costs by up to 90% and latency by up to 85% on long, repeated prompts; real-world savings tend to land below that ceiling but remain substantial, as the case study later in this article shows.3
Context window management is the other lever. Agentic systems that accumulate conversation history without pruning it will see input token counts grow with every step in a workflow. Implement sliding window strategies – summarizing older context rather than retaining it verbatim – and set hard limits on context length per agent session.
Model tier routing: using the right tool for the task
The instinct to route every request to the best available model is understandable. It feels like the safe choice. In practice, it’s one of the costlier habits in enterprise AI operations, because the cost differential between model tiers is large and the performance differential on simple tasks is negligible.
A flagship model is the right choice for complex reasoning, nuanced generation, and high-stakes outputs. It is not the right choice for classifying a support ticket, extracting a date from a document, or checking whether a piece of text meets a formatting rule. For those tasks, a smaller, faster, cheaper model will perform equivalently at a fraction of the cost.
The solution is a routing layer: a lightweight classification step that evaluates the complexity of an incoming request and dispatches it to the appropriate model tier. This doesn’t need to be sophisticated. A simple heuristic – short input, structured output required, no multi-step reasoning – is enough to route the majority of simple tasks away from expensive models. Teams that implement routing see meaningful savings in practice: RouteLLM, an open-source routing framework from UC Berkeley’s LMSYS research group, achieved cost reductions ranging from 35% to over 85% across standard benchmarks while retaining 95% of GPT-4-level performance.4
The routing layer also gives you a natural place to implement cost guardrails: per-user or per-team spend caps, rate limits on premium model access, and fallback behavior when a tier is unavailable or over budget.
Inference compute: the provisioning trap
For teams running self-hosted or cloud-hosted models on dedicated infrastructure, inference compute is often the largest single cost line. It is also the one most prone to systematic over-provisioning.
The provisioning trap works like this: you size your infrastructure for peak load, because the alternative is latency spikes or dropped requests during busy periods. But AI workloads are rarely uniformly distributed. There are predictable peaks – business hours, end-of-month reporting runs, batch processing jobs – and long stretches of significantly lower utilization in between. Provisioning for peak means paying for idle capacity the rest of the time.
Autoscaling addresses this directly, but it requires your inference infrastructure to support fast spin-up and spin-down. GPU instances take longer to initialize than CPU-based services, which limits how aggressively you can scale down. The practical approach is a combination of a modest always-on baseline (sized for off-peak load), autoscaling headroom (sized for predictable demand spikes), and spot or preemptible instances for non-latency-sensitive batch workloads where interruption is acceptable.
Request batching is the other underused lever. Many AI workloads don’t require real-time responses: document processing, content enrichment, nightly report generation. Batching these requests and processing them during off-peak hours reduces peak compute requirements and allows more aggressive use of spot pricing. The barrier to batching is usually architectural rather than technical: systems designed for synchronous request-response patterns need to be adapted to support asynchronous processing queues.
Retry and failure costs: the agentic multiplier
In standard API applications, failure costs are relatively bounded: a failed request is retried once or twice, and the cost of the retry is similar to the cost of the original call. In agentic systems, failure costs have a multiplier effect that most teams don’t fully account for when they first deploy.
An agent that encounters an error midway through a multi-step workflow doesn’t just retry one call, it may restart the entire workflow from the beginning, re-executing every preceding step at full cost. In a workflow with ten tool calls and an average failure rate of 15% per call, the expected number of total calls per successful completion is significantly higher than ten. At scale, this overhead becomes a material cost driver.
The controls here are circuit breakers, retry budgets, and checkpoint-based resumption. Circuit breakers prevent agents from indefinitely retrying a failing tool or API. Retry budgets set a hard cap on the number of retries per workflow, failing gracefully rather than looping. Checkpoint-based resumption – storing intermediate workflow state so that a failure can resume from the last successful step rather than the beginning – is the most effective mitigation but also the most architecturally demanding to implement.
Even without full checkpoint resumption, instrumenting your agentic workflows to track retry frequency and failure rates per step gives you the data to identify which tool integrations are driving disproportionate retry overhead. Often, a small number of unreliable integrations account for the majority of failure-driven cost.
Embedding and vector storage: the quiet re-indexing tax
For any system that relies on a vector database (retrieval-augmented generation, semantic search, agent memory) cost accrues in two places: the embedding calls that convert content into vectors in the first place, and the ongoing storage and querying of the index those vectors live in. Neither shows up on a compute dashboard, which is part of why this driver gets less attention than the other four.
The waste pattern here is re-embedding unchanged documents. Most indexing pipelines run on a schedule rather than in response to change: a nightly job that reprocesses the entire corpus, whether or not anything in it actually changed since the last run. That means most embedding calls on any given cycle are spent regenerating vectors for content that was already correctly indexed. The problem compounds on the storage side, too, since indexes rarely carry a retention policy: vectors for deleted, deprecated, or superseded content stay in the index indefinitely and keep surfacing in retrieval results, adding noise to search quality along with cost to storage.
The fix is incremental indexing: track a checksum or a last-modified timestamp for each source document, and only send it through the embedding model when that value changes. Paired with a TTL or retention policy that automatically expires vectors tied to content that has been deleted or superseded, this turns a full-corpus job into a delta job, one where most runs touch only what is actually new.
Of the five drivers, this is the one least likely to show up on anyone’s radar. It lives in a vector database bill, not a compute bill, and unlike a runaway model tier or an idle GPU cluster, there is no dramatic spike to flag it. It simply grows, month over month, as more content gets embedded and less of it ever gets cleaned up.
Putting it together: the cost visibility layer
The common thread across all five cost drivers is visibility. Most enterprise AI cost problems persist not because they’re hard to fix, but because they’re hard to see. Token spend, model tier distribution, compute utilization, retry rates, and embedding costs sit in different systems, owned by different teams, and no one has a consolidated view of the full picture.
Building a cost visibility layer – even a simple one – changes this. At minimum, it should track: total token spend broken down by application and team, model tier distribution across request types, compute utilization by hour and day, retry rates per agent workflow and per tool, and embedding and storage costs by index.
This doesn’t require custom infrastructure. Most observability platforms can be configured to capture these metrics with modest instrumentation effort. The goal isn’t perfect cost accounting; it’s directional visibility sufficient to identify where the waste is concentrated. In most organizations, a small share of use cases tends to concentrate the bulk of AI cost, and that handful usually contains at least one or two patterns that are straightforward to optimize once they’re visible.
Consider ProjectDiscovery, a security research company whose autonomous testing platform, Neo, runs multi-agent workflows for vulnerability assessments and code review at scale. Its agents relied on system prompts of 2,500+ lines and more than 20,000 tokens each, with a cache hit rate of just 7%. By restructuring prompts to keep static content in a cacheable prefix – and moving dynamic context to the end of the request instead – the team pushed cache hit rates up to 84% and cut overall LLM costs by 59%, reaching 70% in the weeks that followed, with no change to output quality.5 The pattern – one high-volume workflow quietly dominating spend because of an uncompressed, uncached prompt – is common enough to be worth auditing for directly.
The CFO’s questions about AI costs are legitimate and they’re coming whether your team is ready for them or not. Token pricing, inference compute, and model selection are not arcane technical details, they are the unit economics of your AI strategy. Engineering teams that can speak to them clearly, and demonstrate that they have the controls in place to manage them, earn the confidence to scale.
Building cost discipline into AI operations is not about cutting corners. It is about making sure that the value your AI systems deliver holds up at the scale you’re planning to reach.
Notes
1. ICONIQ, “2026 State of AI Report: The Builder’s Economy” (survey of ~305 software executives building AI products, Q2 2026). iconiq.com/growth/reports/state-of-ai-2026. ↩ Volver
2. Statista, “Investment in AI as Share of Revenue by Industry” (survey of 2,360 organizations; published June 9, 2026). statista.com/statistics/1661889. ↩ Volver
Sources Consulted
Anthropic. “Prompt Caching with Claude.” (cited) claude.com/blog/prompt-caching. Anthropic. “Token-Saving Updates on the Anthropic API.” claude.com/blog/token-saving-updates.
CloudZero. “The State of AI Costs in 2025.” Survey of 500 U.S. software engineers and senior managers, March 2025. cloudzero.com/state-of-ai-costs.
ICONIQ. “2025 State of AI Report.” iconiq.com/growth/reports/2025-state-of-ai.
ICONIQ. “2026 State of AI Report: The Builder’s Economy.” (cited) iconiq.com/growth/reports/state-of-ai-2026.
LMSYS Org. “RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing.” July 1, 2024. (cited) lmsys.org/blog/2024-07-01-routellm.
Mavvrik. “2025 State of AI Cost Management Research.” Survey of 372 enterprise organizations, published September 10, 2025. mavvrik.ai/2025-state-of-ai-cost-management-research.
Menlo Ventures. “2025: The State of Generative AI in the Enterprise.” Survey of 300+ executives, April 2025. menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise.
ProjectDiscovery. “How We Cut LLM Cost With Prompt Caching.” April 10, 2026. (cited) projectdiscovery.io/blog/how-we-cut-llm-cost-with-prompt-caching.
SaaStr. “Inference Costs Average 23% of Revenue at AI B2B Companies — How Will You Pay for It?” February 5, 2026. saastr.com/inference-costs-average-23-of-revenue-at-ai-b2b-companies.SaaStr. “The Builder’s Economy: 10 Metrics From ICONIQ’s Newest 2026 State of AI Report.” saastr.com/the-builders-economy-10-metrics-from-iconiqs-newest-2026-state-of-ai-report.
Statista. “Investment in AI as Share of Revenue by Industry.” Published June 9, 2026. (cited) statista.com/statistics/1661889.
