How to Integrate AI into Apps and Save Costs

Key Takeaways:

OpenAI’s Batch API and Anthropic’s Message Batches API both charge exactly half the standard token prices, as stated in the vendors’ official documentation. This is the largest confirmed cost-saving opportunity in AI integration today.
Batch processing is not without limits: OpenAI guarantees completion within 24 hours, and Anthropic limits batches to 100,000 requests or 256 MB, discarding any unfinished work after 24 hours.
Anthropic’s batch endpoint does not accept stream: true, and Amazon Bedrock’s batch inference lacks support for tool calling or structured output. Batch and interactive methods serve different purposes.
Choosing edge inference depends more on hardware capabilities than software design. Transformer workloads are limited by memory bandwidth rather than raw compute power, so using a dedicated NPU is more important than clock speed.
Token pricing varies widely: budget models cost between $0.14 and $1 per million tokens, midrange models range from $2 to $15, and high-end models cost $20 to $75 per million tokens, according to TechTarget’s July 2026 enterprise guide.

The most significant confirmed cost-saving in AI integration is the 50% discount that both major model providers openly offer, which many teams have yet to use. OpenAI’s Batch API documentation clearly states a “50% cost discount compared to synchronous APIs,” and Anthropic’s Message Batches API page confirms all usage is “charged at 50% of standard API prices.” Both vendors state the same figure in their official documentation.

This detail is important because it is verifiable. The version of this article published in May 2026 relied on cost analysis from a source that is no longer accessible. The architectural reasoning in our earlier analysis of AI integration patterns remains valid, but the underlying numbers were not traceable to primary sources. This update reconstructs the five patterns using vendor documentation and published research, adding operational constraints that determine whether each pattern is practical for your use case.

What Changed Since May 2026

Three key developments occurred between the May analysis and now. First, the cost advantage of asynchronous processing became verifiable rather than anecdotal. Both OpenAI and Anthropic publish the 50% discount figure directly, along with the specific limits that apply. Second, the industry’s spending shifted further toward inference. TechTarget’s July 2026 enterprise guide notes that training used to represent most AI computing costs, but that balance “is quickly shifting to inference,” citing McKinsey research on lowering inference expenses.

Edge deployment hardware — Edge Deployment

Synchronous APIs and Streaming

Synchronous request-response remains the default method and the most expensive per token. A client sends a request, waits, and receives a complete response. This method carries the full undiscounted price.

The main refinement to understand is streaming, which does not lower cost but reduces perceived latency. OpenAI provides it as a primary feature in its API documentation. The practical effect is that users see the first tokens arrive while the model continues generating. For interactive applications, time-to-first-token determines user experience more than total generation time. Streaming improves responsiveness but does not reduce expenses, so teams expecting cost savings from it often face disappointment.

The second refinement is background mode, which OpenAI describes as a way to run long tasks “without having to worry about timeouts or other connectivity issues.” You set background to true, receive a response object immediately, and poll its status as it moves through queued and in_progress states. This method offers asynchronous execution but charges standard synchronous prices instead of the 50% batch discount. The trade-off is flexibility at a higher cost.

There is a data-handling detail that surprises some teams. OpenAI’s documentation states that background requests from Zero Data Retention projects run with store=false, and response data is “temporarily stored to disk for roughly 10 minutes to enable asynchronous execution and polling.” If your compliance requires strict zero retention, that ten-minute window is a design constraint to address before deployment.

Asynchronous and Batch Inference

Batch inference offers the confirmed cost benefits. OpenAI’s Batch API accepts a .jsonl file where each line is a request with a unique custom_id, uploads it via the Files API with the purpose set to batch, and returns results when the job finishes. The documentation lists three advantages in order: 50% cost discount, significantly higher rate limits than synchronous endpoints, and guaranteed completion within 24 hours.

Anthropic’s batch process works similarly but with different limits. A Message Batch is capped at 100,000 requests or 256 MB, whichever comes first. Most batches finish in under an hour, and results are available when all messages complete or after 24 hours, whichever occurs first. Batches unfinished after 24 hours expire. Results remain downloadable for 29 days.

The constraints are more important than the discount because they determine which workloads qualify. Anthropic’s batch endpoint explicitly rejects three parameters, and the reasons clarify the model. stream: true fails because batch results return as a single file rather than a stream. The speed parameter fails because fast mode optimizes synchronous latency, which does not apply to asynchronous processing. max_tokens: 0 fails because the temporary cache created during batch processing would expire before a follow-up request could use it.

Amazon Bedrock enforces stricter limits. Its batch inference documentation states that batch inference is not supported for provisioned models and that it “does not support tool calling (function calling) or structured output.” Each input record is processed independently without multi-turn interaction. If your pipeline relies on tool calling or returning schema-bound JSON, Bedrock batch is not suitable regardless of cost savings.

This last point is often misunderstood. Batch processing uses a different execution model with fewer features than synchronous calls with longer timeouts.

Event-Driven and Microservice Patterns

Event-driven architecture separates the request producer from the consumer that processes it. A service emits an event, a broker holds it, and one or more consumers process it independently. For AI workloads, this means inference becomes a subscriber rather than a blocking step in the request path.

This pattern works well with batch processing. A document ingestion service emits an event per document, an aggregator groups them into a window, and a batch job processes the window applying the 50% discount. The event layer manages coordination; the batch layer manages cost. Bedrock even documents EventBridge integration to notify you when batch jobs complete or change state instead of requiring polling.

The trade-off is operational complexity. Event-driven systems need a broker, schema management so producers and consumers agree on payload formats, and observability tools that trace a single logical request across asynchronous steps. A synchronous API call has one failure mode and one log entry. An event-driven pipeline must handle broker availability, consumer lag, poison messages, and duplicate delivery. For workloads processing a few thousand requests daily, a queue and a scheduled job outperform a full event mesh in all key areas, including the often-overlooked cost of engineering time to maintain it.

Edge Deployment

Edge inference runs the model where data is generated, eliminating network latency and cloud API fees. The June 2026 SemiEngineering article explains the demand clearly: developers need to run transformer-capable models on limited devices to protect data privacy, avoid cloud charges, and ensure offline reliability. On-device execution is also becoming necessary to comply with regulations like Europe’s Cyber Resilience Act.

The engineering reality is more complex. Running transformer models with hundreds of millions of parameters on standard CPUs is much less efficient than using dedicated hardware. When AI workloads run on host cores, they consume compute resources needed by other parts of the application. The article describes a Synaptics and Google Research collaboration combining the Synaptics Astra SL2610 processor line with a Coral NPU, which includes a transformer-capable core working alongside a scalar RISC-V core. The reference model is Google’s Gemma 3 270M, an instruction-tuned model with 18 transformer layers.

Note the scale. A 270 million parameter model is roughly 1,000 times smaller than a frontier model. Edge deployment runs a much smaller model on a focused task, and the engineering challenge is deciding which tasks to assign.

Latency and Cost Comparison

Pattern	Latency profile	Cost basis	Key constraint	Source
Synchronous API	Full response time	Standard token pricing	Rate limits and concurrency caps	OpenAI Batch API docs
Streaming	Lower time-to-first-token	Standard token pricing	Does not reduce cost, only perceived latency	OpenAI streaming guide
Background mode	Asynchronous, polled	Standard token pricing	Roughly 10-minute disk retention with store=false	OpenAI background mode docs
Batch (OpenAI)	Within 24 hours	50% of standard prices	One model per input file; no streaming	OpenAI Batch API docs
Batch (Anthropic)	Most under 1 hour, 24-hour cap	50% of standard prices	100,000 requests or 256 MB; expires at 24 hours	Anthropic batch docs
Batch (Bedrock)	Asynchronous, S3 output	See Bedrock pricing	No tool calling or structured output; not for provisioned models	Bedrock batch docs

For model pricing tiers, TechTarget’s July 2026 guide provides ranges to base cost models on: budget models for simple tasks cost $0.14 to $1 per million tokens, midrange models cost $2 to $15 per million, and high-end models for complex tasks and coding cost $20 to $75 per million. The guide also notes that agentic tasks can consume 30 times more tokens than basic chat interactions, which explains many budget overruns that appear to be pricing issues but actually stem from architectural choices.

Routing Decisions That Pay for Themselves

The patterns described are not mutually exclusive. The effective architecture uses a router that sends each request to the least expensive pattern that meets its latency requirements. This routing layer also holds the strongest confirmed cost savings.

AT&T cut AI operating costs by up to 90% on its internal assistant and increased throughput from 8 billion to 27 billion tokens per day, according to VentureBeat’s interview with chief data officer Andy Markus. The key was a routing layer that directed routine tasks to small language models while reserving large models for complex cases, as explained in our analysis of small language model economics. This routing approach, not the choice of model alone, produced the largest cost reduction mentioned here.

A practical routing strategy, ordered by implementation complexity:

Send any request that can tolerate a 24-hour delay to a batch endpoint. The 50% discount requires no new infrastructure, just a file format change and a polling loop.
Send interactive requests to streaming so users see output early, while keeping total generation within budget.
Send long-running reasoning tasks to background mode instead of holding connections open.
Send narrow, high-volume classification tasks to small models, reserving high-end models for work that truly requires them.
Run models on-device only when privacy, offline operation, or regulations require it, accepting the smaller model size limits.

The most costly pattern for teams is using synchronous calls by default because it is the simplest to implement. This default approach results in paying full price on every workload, including those without latency constraints.

Key Takeaways

The 50% batch discount is confirmed in both OpenAI’s and Anthropic’s official documentation and is the largest single cost-saving opportunity without changing models.
Batch endpoints are not direct replacements: Anthropic rejects streaming, and Bedrock batch does not support tool calling or structured output.
Background mode provides asynchronous execution at standard prices, with about a 10-minute disk retention window when store is false.
Event-driven pipelines add overhead for brokers, schema management, and observability that only pays off at higher volumes.
Edge inference runs models roughly 1,000 times smaller than frontier models, so it involves choosing tasks carefully rather than just deployment location.
AT&T’s routing layer cut costs by up to 90% while increasing throughput from 8 billion to 27 billion tokens per day, making routing the highest-impact architectural investment in the stack.

For a CTO preparing board materials, the approach is clear. Measure your monthly token usage, categorize it by latency needs, and apply the batch discount to the portion that can tolerate delay. This calculation is defensible because it uses your actual usage data and published prices rather than vendor benchmarks. The architecture and budget questions are the same, and the solution lies in the routing layer, not just the model choice.

More in-depth coverage from this blog on closely related topics:

Sources and References

Sources cited while researching and writing this article: