AI Inference Cost Trends and Economics

OpenAI cut the price of running its models in half on September 22, 2026, when GPT-6 Luna dropped to $0.10 per million input tokens and $0.50 per million output tokens, while GPT-6 Sol landed at $2.00 and $10.00, per the launch announcement. Those are permanent rates, not promotions, and they sit on top of a two-year collapse in per-token pricing that has reshaped what it costs to build on AI. The paradox that every engineering leader now faces is that token prices are falling at the same time enterprise AI bills are climbing, because the number of tokens consumed per task is exploding faster than the price per token is dropping.

Key Takeaways:

• GPT-6 Luna now costs $0.10/$0.50 per million input/output tokens, a 50% cut from GPT-5.6 Luna, while GPT-6 Sol landed at $2/$10 the same day.
• Gartner forecasts token costs will fall 95% by 2030, yet inference spending per agentic workflow will rise more than fivefold through 2028, the “inference paradox.”
• Prompt caching cuts cached input token costs by up to 90%, which changes unit economics for repeat-call agents with stable system prompts.
• Self-hosting an open-weight model runs around $1 per million tokens versus roughly $50 per million on frontier APIs, with break-even near 5 to 10 million tokens per day.

The Token Price Decline and the 10x Curve

The headline number in AI economics right now is price per million tokens, and that number has been in freefall. Goldman Sachs estimates semiconductor providers are delivering inference cost reductions of 60% to 70% per year per token. Gartner is more aggressive, projecting token costs will fall 95% by 2030, according to Computerworld’s coverage of the firm’s Tokenomics Model. Compounded, that is roughly a 10x cost decline at constant capability level every couple of years, the kind of curve that previously only existed in storage and bandwidth.

The GPT-6 Sol and Luna launch makes that curve concrete. Against OpenAI’s published GPT-5.6 rates, Sol is exactly 50% cheaper in both directions, and Luna is 50% cheaper on input and 58.3% cheaper on output, according to VentureBeat’s launch coverage. Anthropic has been pushing the same lever through prompt caching, which the company positions as the single biggest cost saver for agentic and coding workloads, and its latest Claude Opus 5 release was positioned around “near Fable performance at half the price”.

What changed structurally is that the industry’s center of gravity shifted from training to inference. Gartner’s August 2026 forecast found that AI-optimized infrastructure-as-a-service will reach $42 billion in 2026, and for the first time $23.3 billion of that flows to inference versus $19 billion to training. Inference is now 55 cents of every AI cloud dollar. That is why labs are fighting a price war on serving costs rather than on benchmark leaderboards.

Input-Output Asymmetry and Prompt Caching

Token pricing is not symmetric, and the asymmetry is where most budgets leak. Output tokens are priced 5x to 10x higher than input tokens across nearly every provider because generation is sequential and compute-heavy, while input processing can be batched. GPT-6 Luna charges $0.10 for input but $0.50 for output; GPT-6 Sol charges $2.00 input against $10.00 output. A model that reasons at length, retries, or emits long explanations multiplies its own cost on the output side.

Prompt caching is the single most underused lever for closing that gap. OpenAI’s GPT-6 Sol and Luna launch included caching upgrades that cut cached input token costs by 90%. Anthropic’s cache reads, which the company says make up most of the cost for agentic and coding workloads, are priced at a steep discount to regular input. Moonshot’s Kimi K3 prices cached input at a fraction of its list rate, and the company reports cache hit rates above 90% on coding workloads.

The reason this matters for agents specifically is that re-sent context is the dominant cost driver. Stanford Digital Economy Lab research cited in Gartner analysis found that re-sent context, meaning system prompts, tool definitions, and session history reprocessed on every call, accounts for roughly 62% of total agent inference costs. A coding agent that re-reads a 100,000-token repo on every tool call is paying for that context repeatedly. Structuring the prompt so that the stable prefix is cacheable turns that 62% cost into a near-zero line item on cache hits.

A Worked Example: 1M MAU at 5 Calls a Day

The gap between naive and optimized deployment shows up fastest in a real workload. Take a product with 1 million monthly active users, each making 5 calls per day, averaging 1,200 input tokens and 300 output tokens per call. That is 150 million calls per month.

The arithmetic: 150 million calls x 1,200 input tokens = 180 billion input tokens, and 150 million x 300 output tokens = 45 billion output tokens per month. Multiply by each tier’s rates and the spread is a 100x difference between Luna and the frontier tier for the same nominal task volume.

Two caveats change the picture. First, if 60% of those input tokens are a stable, cacheable prefix, the effective input cost drops by up to 90% on that portion, cutting the Luna bill toward roughly $15,000 and the Sol bill toward $300,000. Second, if the workload is agentic rather than single-turn, the token count per call explodes. Gartner predicts inference costs per agentic workflow will rise more than fivefold through 2028 as agents reason, replan, and call other agents in the background. The same 1M MAU product, if it chains many model calls per task instead of one, moves from a $40,500 problem to a seven-figure one regardless of which tier it routes to.

When Self-Hosting Wins

The open-weight alternative has become genuinely competitive at the frontier. Kimi K3, a 2.8-trillion-parameter Mixture-of-Experts model Moonshot AI released in July 2026, scores within a few points of the best closed systems on independent tests, and its weights are downloadable. CoreWeave’s analysis puts the serving math bluntly: typical latest-generation frontier models cost about $50 per million output tokens, while serving a right-sized open model on your own GPUs runs closer to $1 per million tokens. Cost analyses this year put break-even for self-hosting near 5 to 10 million tokens per day.

That break-even is a use threshold, not a sticker price. Self-hosting only wins when the team can keep GPUs busy. The hardware side is not getting cheaper: Nvidia H100 one-year rental contracts hit $2.35 per GPU-hour in March 2026, up nearly 40% from $1.70 in October 2025, according to SemiAnalysis. A 2.8-trillion-parameter model quantized aggressively still needs on the order of hundreds of gigabytes of accelerator memory before runtime overhead, which is a multi-GPU deployment, not a laptop project.

The honest trade-off is this: self-hosting wins when three conditions hold simultaneously. First, sustained volume above the 5-to-10-million-token daily threshold so the fixed GPU cost amortizes. Second, a workload with stable context that benefits from caching and distillation, since open weights let you quantize, batch, and distill a large teacher into small task-specific students, levers a closed API does not expose. Third, a data-control or latency requirement that a shared API cannot meet. Below that volume, or for bursty traffic, the API is cheaper because the provider absorbs the idle-GPU risk. As we covered in our Opus 5.5 review, the same-day Sol launch turned the competitive question from “who has the best benchmark” to “who has the lowest cost per completed task.”

Where the Cost Floor Is

The cost curve does not run to zero. The floor is shifting from raw FLOPs to physical constraints: power delivery, cooling, memory bandwidth, and the margin infrastructure owners need to earn. Gartner’s procurement guidance makes the point directly: inference is a memory-bandwidth-bound workload, not a compute-bound one. During generation the model loads weights and KV cache data from high-bandwidth memory on every step, so the limiting factor is how fast data moves, not how fast the chip multiplies. Nvidia’s Blackwell B200 delivers roughly 10x lower cost per token than Hopper primarily by lifting memory bandwidth, which Nvidia’s own published analysis frames as “tokens per watt” determining AI factory revenue.

There is also structural tension in pricing. Token prices are falling while GPU rental prices are rising. CME Group plans October 5 futures on Nvidia H100 and B200 rental prices, each contract covering 730 GPU-hours, a signal that the market now treats GPU-hours as a tradable commodity with expected price appreciation. If per-token revenue falls faster than per-GPU-hour cost falls, the labs and neoclouds absorbing that squeeze will eventually stop subsidizing inference, which is exactly the margin-squeeze scenario Forbes flagged in July 2026.

The practical floor for enterprise is therefore not token price but the cost-per-completed-task number, and it is specific to each workload. Gartner’s Tokenomics Model maps per-task costs across workflow types, from basic single-step tasks at the low end to planning and learning workflows at the high end, an 8x to 10x spread driven by how many tokens each task type consumes, not by the model’s list price.

What This Means for Engineering Strategy

The teams that win in 2026 are treating model selection as a routing problem, not a loyalty decision. The pattern that recurs across every serious deployment is inference tiering: send routine, high-volume traffic to the cheapest model that clears the quality bar, reserve frontier models for genuinely hard tasks, and cache every stable prefix. Gartner’s advice to enterprises is to block agents from invoking frontier models by default for simple tasks, and to treat each model release as “a new car losing value on day one.”

The numbers behind that advice are stark. Uber blew through its entire 2026 AI coding budget by April, with the company’s CTO admitting the Claude Code budget was “blown away already,” according to TechCrunch’s reporting on the token bill coming due. Box CEO Aaron Levie warned that agentic AI costs are escalating as agents move beyond engineering. These are architecture problems, not pricing problems. The cost per token fell, but tokens per task rose faster.

My forecast: by mid-2027, GPT-6 Luna-class pricing (sub-$0.15 input) will become the default tier for routine enterprise workloads, and the frontier tier will be reserved for a shrinking share of tasks as routing and distillation mature. The company that measures cost per completed task, caches aggressively, and self-hosts only above the use threshold will run the same product at a fraction of the cost of a competitor that defaults every call to a frontier model, and that gap is now a moat.

Related Reading

More in-depth coverage from this blog on closely related topics:

• How to Train a 4B Model
• Opus 5.5 Update: Features and Review
• Moonshot V2 Launchpad: Kimi K3 Review
• Dart Programming Language Gets Stable
• Grok 4.7 Features and How to Use

Sources and References

Sources cited while researching and writing this article:

• OpenAI cuts GPT-6 Sol and Luna API prices by 50%
• inference cost reductions of 60% to 70% per year per token
• AI inference is getting cheaper, but your agents are getting more expensive
• OpenAI releases GPT-6 Sol and Luna models, slashing API costs 50% or more
• Anthropic’s new Claude prompt caching will save developers a fortune
• Gartner Marks First Year Inference Spending Beats AI Training: 55 Cents of Every Cloud Dollar
• Inference Cost: Why Kimi K3 Isn’t Free
• according to SemiAnalysis
• Tokens per Watt Determines AI Factory Revenue as Power Constraints Tighten
• The token bill comes due: Inside the industry scramble to manage AI’s runaway costs