I Cut AI Token Use by 69% Without Breaking the Delivery Promise

A small model call became expensive only after multiplication.

The workflow processed millions of items. The first implementation sent each item independently:

Input tokens: 78
Output tokens: 11
Total tokens: 89

Eighty-nine tokens was not the problem.

Millions of independent calls were.

The obvious cost-saving option was provider-side asynchronous batching. It lowered the token price, but it could take up to a day to complete. The workflow already had a daily delivery promise. A cheaper call that delayed downstream processing was not a clean optimization.

I changed the application instead.

By grouping bounded sets of items and removing repeated instructions, I reduced token use by about 69% while preserving the delivery window, validating every item, and keeping failures recoverable at item level.

The Delivery Promise Was the Constraint

The workflow looked like this:

input data
  -> preliminary processing
  -> hourly aggregation
  -> AI decision
  -> daily post-processing
  -> delivery

The AI step did not exist in isolation. Its latency affected every downstream stage.

That made the real requirement:

Reduce operating cost without moving uncertainty into the daily delivery promise.

Provider-side batching changed the timing contract. It may still be useful in workflows with loose deadlines, but it was the wrong first move here.

Reduce Work Before Negotiating Price

The original path repeated the same instructions for every item.

For 50 items:

89 tokens × 50 = 4,450 tokens

I grouped the items into one bounded request and compressed the repeated context:

Input tokens: 975
Output tokens: 399
Total tokens: 1,374

The reduction was:

(4,450 - 1,374) / 4,450 ≈ 69%

There are two distinct optimizations:

• provider batching lowers the price of tokens
• application aggregation removes unnecessary tokens

Reducing unnecessary work was the better first step because it preserved the existing workflow shape.

The Cheaper Path Changed the Failure Shape

One item per request has a small blast radius.

Fifty items per request changes that.

The aggregated design introduced new risks:

• one malformed item could disrupt the group
• weak separators could let one item influence another
• the model could omit an item
• the model could duplicate an item
• a valid response could attach a decision to the wrong input
• compression could remove a fact needed for the correct decision
• one provider failure could affect many items
• retaining only the compressed prompt could weaken auditability

The token count improved.

The reliability model had to change with it.

Define the Invariants

The core invariant was:

Compression may remove repeated tokens, but it must not remove facts required for the same business decision.

I translated that into checks the application could enforce:

• every input has a stable item ID
• every input receives exactly one output
• unknown output IDs are rejected
• duplicate output IDs are rejected
• malformed items do not block valid items
• failed items can be retried independently
• batch size remains bounded
• the new path can be compared with the original path
• prompt and policy versions remain attributable

The model did not become reliable because the prompt was shorter.

The surrounding software made the aggregated output inspectable.

Use Stable Item Boundaries

Each input carried an explicit identity:

ITEM_ID: 7f2a
INPUT: ...

The response used a strict structure:

{
  "item_id": "7f2a",
  "decision": "approve",
  "reason_code": "RULE_4"
}

The application validated:

input IDs == output IDs
count(input IDs) == count(output IDs)
no duplicates
no unknown values
all required fields present
all decision values allowlisted

A syntactically valid model response was not enough.

The response had to preserve one-to-one correspondence with the input set.

Keep the Batch Bounded

If 50 items were cheaper, 5,000 might appear better.

But efficiency is not the only variable.

A larger batch increases:

• retry cost
• investigation cost
• memory and context pressure
• cross-item influence
• the number of decisions affected by one malformed response

The right batch size is not the largest number accepted by the provider.

It is the largest unit the system can fail, inspect, retry, and recover without creating an operational problem.

Hourly aggregation provided a useful containment boundary:

One failure affects at most one bounded group, not the entire day.

Compare Against the Existing Path

A response can be valid JSON and still represent behavioral drift.

Before replacing the original path, I ran both paths against the same inputs:

same input
  -> original path
  -> aggregated path
  -> compare decisions by item ID

The rollout measured:

• disagreement rate
• missing-output rate
• duplicate-output rate
• malformed-response rate
• retry rate
• latency
• tokens per item
• changes in decision distribution

The comparison made the optimization falsifiable. A lower token count did not automatically justify rollout.

Recover at Item Level

A failed group should not force the entire group to be replayed if only one item is invalid.

The recovery path was:

validate group
  -> accept valid items
  -> isolate failed items
  -> retry failed items independently
  -> record original and retry outcomes

This preserved the cost benefit for valid items while keeping one malformed item from blocking the hour.

The system also retained enough evidence to answer:

• which prompt version processed the item
• which group contained it
• which decision was returned
• which validation failed
• whether it was retried
• whether the original and aggregated paths disagreed

The Result

The final design delivered:

• about 69% fewer tokens
• no next-day dependency
• no downstream workflow rewrite
• bounded groups
• strict item-level validation
• measurable disagreement
• item-level recovery
• explicit privacy and correctness review

The important result was not simply a cheaper model call.

The workflow became cheaper without hiding its new failure modes.

Optimization Checklist

Before accepting a cost reduction, ask:

1. Which cost was reduced?
2. Which cost may have moved elsewhere?
3. Did the latency contract change?
4. Did the privacy boundary change?
5. Did the blast radius increase?
6. Can every unit be matched to one output?
7. Can failed units be retried independently?
8. Can the new path be compared against the old one?
9. Can a past decision still be explained?

A production optimization is complete only when the saving and the moved risk are both visible.