This is the pattern we see repeatedly. A model that performed elegantly in testing quietly multiplies its consumption in the wild, through longer conversation histories, retrieved documents, retries, and agentic workflows that chain eight model calls to do what once appeared to be a single operation. Token spend is not a line item that stays still. It compounds.
The goal of this article is not to convince you that AI is expensive or that it is cheap. Both framings are misleading. The real question is whether your organisation has the visibility and control to spend deliberately, and right now, most do not.
Key takeaways
Input and output tokens are priced differently for architectural reasons, not arbitrarily: output generation is sequential and compute-intensive, typically three to five times the price of input, and often more, so systems producing long responses carry a fundamentally different cost profile than those reading long prompts.
Not every task needs the newest, most capable model; routing 70 to 80 per cent of queries to cheaper models and reserving frontier capability for genuinely complex work is an engineering decision with quantifiable financial consequences, not a compromise on quality.
The highest-return cost lever available today is usually the simplest to implement: prompt caching alone can cut a stable system prompt's cost by roughly 90 per cent, and it is consistently underused.
Input AI tokens vs output tokens: why the asymmetry matters more than the headline price
Input AI tokens vs output tokens: why the asymmetry matters more than the headline price
Providers publish pricing in pairs: an input rate and an output rate, denominated per million tokens. The output rate is consistently higher: typically three to five times the input price, and sometimes more. Every Anthropic model in the table below sits at exactly 5x, and the budget models run higher still. Most teams notice this and shrug. They should not.
The reason for the asymmetry is architectural. Input tokens are processed in parallel across GPU cores. The model reads your prompt, your system instructions, and any retrieved context all at once, in a single forward pass. It is computationally efficient. Output tokens are generated one at a time: each new word requires a complete forward pass through the entire model before the next can be produced. That sequential dependency is expensive, and the cost accumulates with every character the model writes.
The practical consequence is that the same headline model price behaves very differently depending on what you are building. A system that reads long documents and replies briefly has a fundamentally different cost profile than one that generates long, detailed outputs. A summarisation pipeline, a classification service, a routing agent: these are input-heavy and relatively cheap to run. A code generation assistant, a long-form writing tool, or any workflow that produces extended, structured responses is output-heavy and priced accordingly.
Most business cases for AI deployment are built on a back-of-envelope estimate that ignores this distinction entirely. The teams that build correctly priced cases measure their actual input-to-output ratio per request type, in staging, before they commit to architecture. Everyone else finds out on the invoice.
Read more about AI delivery and cost governance:
What determines AI token cost? The key factors
What determines AI token cost? The key factors
Token price is just the starting point. The real cost of running a model in production is shaped by several variables that interact in ways that are not always obvious.
Model tier and provider set the floor. Frontier models, meaning the most capable and most recent releases from Anthropic, OpenAI, or Google, carry the highest per-token rates. Mid-tier and distilled models are cheaper by an order of magnitude. The choice of model is therefore not just a quality decision; it is a financial architecture decision, and the two should be made together.
Context window size is one of the most underappreciated cost drivers. Every token in your context window, including your system prompt, conversation history, retrieved documents, and tool definitions, is billed as input on every single request. A 4,000-token system prompt sent with 10,000 daily requests adds 40 million input tokens per day, or roughly 1.2 billion per month, before a single user query is included. Organisations that have not instrumented their prompt overhead are routinely surprised to discover that 30–40 per cent of their token budget is consumed by structural scaffolding rather than actual user content.
Task type matters in ways that do not always map neatly to model capability. Tasks that require extended chain-of-thought reasoning burn tokens differently from pure generation or classification. Some model families charge a premium for extended thinking modes; others include it implicitly. If you are enabling reasoning features without understanding their token cost profile, you are operating blind.
Deployment method creates a fork in cost structure. API access from a provider like Anthropic, OpenAI, or Google is the default: you pay per token, at published rates, with the provider managing infrastructure. Self-hosted models shift the cost to compute, covering GPU instance hours, memory, and serving infrastructure. Self-hosting is almost never cheaper at low to moderate volumes, but the economics can shift at very high throughput, or when data residency requirements make third-party API access impossible.
Caching eligibility is a lever that many teams simply forget to pull, despite it being one of the fastest wins available. We will come to it in detail shortly.
AI Readiness Assessment
Gain a clear view of how prepared your data is to support and scale AI initiatives in your organisation.
How AI model pricing is structured
How AI model pricing is structured
Providers group models into three broad pricing tiers, and rates shift monthly across providers, so absolute numbers age quickly.
- Budget models, meaning compact or distilled versions, sit at the low end.
- Mid-tier models offer the balance most production workloads actually need.
- Frontier models, the most capable and most recently released, sit at the top.
The spread between the cheapest and most capable tier typically runs in the range of 20 to 30 times on both input and output pricing, though this gap compresses over time as competition between providers increases. A workload that would cost a few dollars a month on a budget model can cost several hundred dollars on a frontier model for equivalent volume. The decision about which tier handles which task is not a quality judgement alone. It is a cost decision with material financial consequences.
Model tiers and the quality-cost trade-off: when is a cheaper model the right engineering decision?
Model tiers and the quality-cost trade-off: when is a cheaper model the right engineering decision?
There is a recurring assumption in AI adoption conversations that the newest, most capable model is the correct default. It is not.
Every new model is designed to solve problems that previous models could not. That is its explicit design brief, and it is also why assuming the latest release is the right tool for every task is a mistake worth examining carefully. Not everything is a hard task. Writing a standard article, processing a well-structured form, classifying a support ticket: these are not problems that require the most capable model available. They require a model that is good enough.
The engineering principle that follows is straightforward: route tasks by complexity, not by default. The production systems we see working well in cost-conscious organisations direct 70–80 per cent of queries to faster, cheaper models, and reserve frontier models for the 10–20 per cent of tasks that genuinely require their capability, whether that is complex reasoning, multi-step code synthesis, or ambiguous edge cases that smaller models demonstrably handle poorly.
This tiered routing is not just a budget measure. It also reduces latency for the majority of requests, since lighter models respond faster, and it forces engineering teams to think explicitly about what each model is actually being asked to do, which is a discipline that pays dividends in quality as well as cost.
Multi-provider routing is increasingly practical. AWS Bedrock and Azure AI Studio both offer access to models from multiple providers through a single API surface. Routing a batch of simple classification tasks to a budget model from one provider while sending complex reasoning tasks to a frontier model from another is architecturally feasible today, and the cost optimisation is real.
Growing competition, not only from established US providers but increasingly from Chinese and European model developers, is accelerating this. The number of viable, credible model options available on managed marketplaces is expanding.
How to control AI token cost?
How to control AI token cost?
Token spend without governance is not a technical problem. It is an organisational one. Before optimising individual API calls, the more important question is: who owns the token budget, what visibility do they have, and how are limits enforced at runtime rather than discovered at month-end?
The organisations that control AI spend well share a few common practices:
- Instrument every significant workflow at the point of execution: log token counts per request type, per agent, per team, before any cost crisis forces you to.
- Set per-application and per-team consumption limits: trigger alerts or throttle requests before bills spiral, rather than conducting a forensic investigation into last month’s charges.
- Distinguish between the cost of a request going right and the cost of a request going wrong: retries, hallucination-induced cascades, and error-handling paths often account for a surprising fraction of production spend.
The other piece that is frequently missing is a clear owner. AI inference cost tends to fall in the gap between the data team, the product team, and finance. None of them takes full accountability. The result is that token spend is reviewed quarterly, reactively, and usually too late to inform the architectural decisions that caused it.
Enhancing festival planning with an AI tool that generates lineup
proposals in under 20 seconds, at near-zero operational cost
Five cost optimisation levers that production teams can use
Five cost optimisation levers that production teams can use
Prompt caching. A reused portion of your prompt, typically your system instructions, static context, or tool definitions, can be stored server-side so that subsequent calls read it from cache at a fraction of the standard input rate. Anthropic charges 10 per cent of the normal input rate on cache hits; OpenAI charges 50 per cent.
For a production application with a consistent 4,000-token system prompt running 10,000 daily requests at a 90 per cent cache hit rate, enabling caching reduces that prompt’s daily cost from roughly $120 to about $25. The cached reads cost a tenth of the standard rate, while the 10 per cent of cache misses still pay full price plus a small write premium. That is a saving of nearly $3,000 per month from a single configuration change. This is consistently one of the highest-return levers available, and it is consistently underused.
Tiered model routing. As outlined above: classify tasks by complexity before invoking a model, and route accordingly. The engineering cost of building routing logic is usually recovered within weeks of production deployment at meaningful scale.
Context window discipline. Every token in context is billed. Keeping conversations tightly scoped, with fewer RAG chunks, compressed system prompts, and cleared conversation histories after task completion, directly reduces input spend. Setting up a conversation so that responses are concise and purposeful is not about degrading quality. It is about eliminating waste. If the model is producing 800 tokens when 200 would have served the user equally well, you are paying for 600 tokens of nothing. For larger projects, structuring instructions hierarchically, with global rules that apply across all tasks and localised instructions that activate only for specific workflows, is one of the most effective ways to reduce redundant context without sacrificing capability.
Batch API for asynchronous workloads. Most providers offer a batch processing tier at a 50 per cent cost reduction for requests that do not require a real-time response. Nightly report generation, bulk data enrichment, document processing pipelines, and offline evaluation jobs are all candidates. If your architecture treats every request as interactive, you are paying real-time rates for work that could comfortably wait.
Output length constraints. Instructing the model explicitly to be brief, and specifying a target format, is the simplest and most overlooked lever available. Language models will, by default, produce thorough, comprehensive responses. In many production contexts, that is not what the application needs. Constrain output length through the system prompt, through structured output requirements, or through post-processing truncation. The cost reduction can be significant, particularly in agentic workflows where model outputs feed into subsequent steps.
Building a token budget: from billing review to execution governance
Building a token budget: from billing review to execution governance
There is a meaningful difference between monitoring what AI has cost and governing what AI is permitted to cost. Most organisations today do the former: they examine the invoice, understand the breakdown, and carry that understanding into the next planning cycle. What they are not yet doing is enforcing token budgets at runtime, setting hard or soft limits per application, per agent, and per team that are checked before inference is invoked.
This distinction matters because AI workloads, unlike most software costs, are non-linear and difficult to predict from first principles. A single misconfigured agentic pipeline, or a sudden spike in user activity, can exhaust a month’s budget in hours. Reactive monitoring catches this after the fact. Proactive governance prevents it.
The emerging discipline for this is sometimes called AI FinOps: applying the governance frameworks that cloud FinOps developed for infrastructure spend to the specific characteristics of AI inference spend. Cloud FinOps gave us reserved instances, savings plans, rightsizing recommendations, and cost allocation by team. AI inference needs its own equivalents:
- Per-model budget envelopes: spend caps set at the model level, not just the account level
- Task-type routing policies: rules that decide which model tier handles which kind of request
- Real-time spend dashboards with automated alerts: visibility before the invoice, not after
- Periodic architecture reviews triggered by cost thresholds: reviews that happen when spend crosses a line, not on a fixed billing calendar
Token optimisation is moving from being an afterthought to a dedicated function. We are already in a world where companies are actively reviewing how their AI spend is structured, which models are used for which tasks, which providers, and what the governance looks like. That is AI FinOps by any other name, and it will be one of the more consequential disciplines in the next few years.
At Future Processing, we are already working with clients at this intersection of AI architecture and cost governance. The engineering disciplines involved, covering instrumentation, cost allocation, policy enforcement, and architectural review, are not novel. What is new is applying them to a cost structure that compounds unpredictably and resists intuitive estimation.
AI-powered delivery
Case study
Reducing job listing processing time by 66% through an AI-powered solution built on AWS
Strategic takeaway for decision-makers
Strategic takeaway for decision-makers
The single most valuable action right now is instrumentation. Before optimising anything, measure what you are actually spending: per workflow, per request type, per agent. Most teams are working from estimates. The gap between estimates and reality, once measured, almost always justifies the effort.
From there, the sequencing matters. Prompt caching offers near-immediate ROI and requires minimal architectural change. Tiered model routing requires more design work but produces the largest long-term savings. Context window discipline and output constraints are ongoing practices, not one-time fixes.
The teams that build that governance framework now, before their AI workloads reach material scale, will have a significant advantage over those that instrument after the fact. The cost of getting this wrong is not just a budget overage. It is the accumulated distortion of every architectural decision made without accurate cost information.
FAQ
How do RAG, agents, and context windows inflate your real token spend?
RAG injects retrieved documents into every query as additional context, typically adding three to five times more tokens per call than a simple prompt. An agent workflow compounds this further by triggering five to thirty model calls per user task, each carrying its own system prompt, tool definitions, and accumulated history. The result is that a user interaction that appears to involve a single model call may actually involve dozens, each billed at full token rates.
The most instructive illustration of this is agentic browser automation. When a model is given the task of filling in a web form, it must first read the entire page content, identify the relevant fields, determine the correct values, and execute each input, often across multiple steps. I observed this directly while testing Claude in Chrome, Anthropic’s browser agent. Watching the tokens disappear on a single webpage was instructive. The agent has to process the whole page to understand it, identify the fields, and decide what to enter. That is a completely different order of magnitude compared to asking the model a direct question and getting an answer. The same information exchange, structured differently, could cost a fraction of the price.
What is prompt caching and how much can it save?
Prompt caching stores a reused portion of your prompt, typically the system instructions or static context, server-side so that subsequent calls read it from cache at a fraction of the normal input price. Anthropic charges 10 per cent of the standard input rate on cache hits; OpenAI charges 50 per cent.
For a production application with a consistent 4,000-token system prompt running 10,000 requests per day, enabling caching at a 90 per cent hit rate reduces that prompt’s daily cost from roughly $120 to about $25, a saving of nearly $3,000 per month from a single configuration change.
How do you calculate AI token cost for a product?
Use this formula: monthly cost = (daily requests × average input tokens × input price per million + daily requests × average output tokens × output price per million) × 30.
The step most teams skip is measuring actual token counts per request type in staging, rather than estimating from word count. System prompts, conversation history, and RAG context routinely push real token usage three to five times above the naive estimate. Add a buffer of 1.5 to 2 times the result to account for retries, error handling, and volume growth in the first quarter of production.
Why are output tokens more expensive than input tokens?
Input tokens are processed in parallel across GPU cores: fast, efficient, and computationally cheap. Output tokens must be generated sequentially. Each word requires a full forward pass through the entire model before the next can begin, which makes generation three to five times more compute-intensive than reading. A system that returns long responses costs fundamentally more than one that reads long documents and replies briefly.
Does prompt length significantly affect AI costs?
Yes, and the effect compounds quickly. A 2,000-token system prompt sent with every message across 10,000 daily conversations adds 20 million input tokens per day before any user content is included. RAG pipelines typically inject a further 1,000 to 5,000 tokens per query on top of that.
Teams that instrument prompt length before deploying to production routinely find that 30–40 per cent of their token budget is consumed by structural overhead rather than the actual user query. That overhead is addressable, but only if you measure it first.
