Most AI agents are built with a very expensive form of amnesia.
Every time an AI agent takes another step, it is handed the same system prompt, the same tool definitions, the same company policies, the same guardrails and often the same 20,000-token wall of context.
Then companies pay the model to process it all over again.
When the AI bill arrives, everyone starts discussing downgrading the model. Use less thinking. Reduce the context window. Move from Opus to Sonnet. Move from Sonnet to something cheaper.
All reasonable options.
But there is another lever that I think many companies have barely understood: caching.
ProjectDiscovery recently published a breakdown of how it increased the cache-hit rate of Neo, its multi-agent security platform, from 7% to 84%. Its overall LLM costs fell by 59%. In its most recently measured ten-day window, the savings had reached 70%. It served 9.8 billion tokens from cache.
One especially large task ran through 67.5 million input tokens with a 91.8% cache-hit rate.
Another task processed a similar volume with only a 3.2% hit rate.
The second cost roughly 60 times more.
That sent me down a rabbit hole because, until recently, I thought caching was fairly binary. You turn it on. You save money.
Apparently, no.
Your cache can be enabled and still be doing almost nothing. It can even cost more than sending ordinary input if you keep writing content that nobody reads again.
So, let us play a game.
I will give you a statement. Decide whether it is true, false or, occasionally, the annoying but honest answer: it depends.
Each answer will tell us something about whether your agent is caching properly.
Prices and product behaviour mentioned in this article were checked on July 15, 2026.
A 30-second explanation of what is being cached
Before starting the game, here is the technical foundation.
When an LLM transformer processes your prompt, it builds internal representations called key and value tensors. Collectively, these are often called the KV cache.
Imagine giving a research assistant a 50-page company brief.
Without caching, the assistant rereads all 50 pages before answering every question.
With caching, the assistant reads the brief once and keeps structured notes from that reading. The next question can begin from those notes instead of page one.
The model still generates a fresh answer. It is the repeated reading of the input that gets skipped.
This is also why exact ordering matters.
A token is interpreted in relation to the tokens that appeared before it. Change something near the beginning and the internal states calculated after that point may no longer be reusable.
You have not changed one isolated sentence. You have changed the foundation on which the remaining prefix was processed.
Statement 1: Once prompt caching is enabled, your AI bill goes down
False.
This was my first misunderstanding.
Turning on caching gives your system the opportunity to save money. It does not guarantee that it will.
To understand why, we need to look at the four ways a token can appear on your bill.
Number one, regular input. The model processes a token normally at the full input price.
Number two, a cache write. The model processes the token and stores the resulting internal state so that it might be reused. On Anthropic, a five-minute cache write currently costs 1.25 times the ordinary input rate. A one-hour write costs twice the ordinary input rate.
Number three, a cache read. The stored state is reused instead of recomputed. This currently costs 10% of the ordinary input rate.
Number four, output. The model generates something new. Prompt caching does not discount this because the output did not exist before.
A cache write is more expensive than normal input. It only becomes worthwhile when later requests read what you stored.
Imagine that you pay to reserve a parking space for an hour. That is useful if you return to it. If you keep reserving spaces in different buildings and never revisit any of them, you have not optimised parking. You have simply added reservation fees.
This is why cache-hit rate matters.
At a 7% cache-hit rate, ProjectDiscovery was still processing almost everything fresh. Moving dynamic content out of its reusable prefix increased the rate to roughly 74% in one deployment.
Caching was technically present before the change. But it was economically optamised.
Statement 2: A one-hour cache is always cheaper than a five-minute cache
False. It depends on your traffic.
Anthropic’s default cache lifetime is five minutes. Every successful hit refreshes that lifetime at no additional write cost.
If the same shared prompt is being hit every minute, a five-minute cache can remain warm all day. In that situation, paying the larger one-hour write premium may be unnecessary.
Now imagine a shared enterprise agent with irregular usage:
Customer A uses it at 9:00.
Customer B arrives at 9:08.
Customer C arrives at 9:21.
The system prompt has not changed, but each gap is longer than five minutes. The cache repeatedly goes cold, so you keep paying to write the same prompt again.
A one-hour TTL absorbs those gaps.
There is a simple way to see the trade-off. A five-minute write costs 1.25 times the base input price. If the same static prefix has to be written twice within an hour, you have paid 2.5 times the base rate. One one-hour write would have cost twice the base rate.
The one-hour cache has already become cheaper.
This makes TTL selection a forecast about user behaviour.
Use five minutes when requests arrive continuously enough to refresh the cache.
Consider one hour when a valuable prefix is reused less frequently than every five minutes but more frequently than every hour, or when one agentic run itself may take longer than five minutes.
Anthropic and AWS both recommend looking at those traffic patterns rather than assuming that the longer cache is automatically better.
One more detail: different parts of the request may deserve different TTLs.
A shared system prompt used by hundreds of customers may deserve one hour.
A rapidly growing conversation history may need only five minutes.
That brings us to manual caching.
Statement 3: Automatic caching means your engineers no longer need to think about prompt structure
False.
Automatic caching is an excellent default. It has also become more capable than I initially realised.
On Anthropic’s API, a developer can place one top-level cache control field on the request. The breakpoint then moves forward as the conversation grows. Automatic caching can use either a five-minute or one-hour TTL, and it can be combined with explicit breakpoints.
So why bother with manual caching?
Because the infrastructure can detect matching token sequences. It cannot reliably infer the architectural intent behind every part of your prompt.
It does not inherently know that:
the system constitution should remain stable for six months;
working memory changes after every tool call;
the customer ID changes for every user;
a policy document changes once per quarter;
today’s date should be frozen for one task;
one section deserves a one-hour TTL while another deserves five minutes.
Automatic caching looks for reusable prefixes. It does not redesign a badly ordered prompt.
Anthropic’s own documentation now calls out this trap. If automatic caching places its breakpoint on a final block that changes on every request, such as a timestamp or per-request context, the prefix hash changes. You may repeatedly create cache entries without receiving useful hits.
Explicit caching gives an engineering team control over where reusable sections end.
Anthropic currently allows up to four breakpoints. Those can separate content that changes at different frequencies and can mix one-hour and five-minute TTLs.
There is no universal winner here.
For a normal multi-turn assistant, automatic caching is a very sensible place to start.
For an enterprise agent containing shared tool definitions, static policies, changing user context, working memory and long tool loops, explicit breakpoints may produce much better economics.
ProjectDiscovery needed the second approach. Its platform was too dynamic and too large for one automatic boundary to express what the system needed.
Statement 4: A timestamp near the beginning of a prompt only invalidates the timestamp
Very false.
This is probably my favourite caching mistake because it looks so harmless.
Imagine this order:
CURRENT TIME: 10:32:41
SYSTEM INSTRUCTIONS 8,000 tokens
TOOL DEFINITIONS 6,000 tokens
COMPANY POLICIES AND REFERENCE MATERIAL 20,000 tokens
CURRENT USER REQUEST 42 tokens
On the next request, the current time becomes 10:32:42.
For prefix caching, the exact token sequence near the beginning changed. Everything after that point was processed in relation to the old prefix and can no longer be treated as an identical cached continuation.
One second can sabotage tens of thousands of otherwise stable tokens.
You should move that dynamic material to the tail of the prompt.
The useful ordering is easier to understand when you draw it:
BEGINNING OF REQUEST
1. Stable tool definitions
2. Stable system instructions and guardrails
3. Stable policies, examples and reference material
CACHE BREAKPOINT
Often suitable for a longer TTL
4. Session-specific context
5. Working memory
6. A date or time frozen for this task
7. Latest tool results
8. Current user request
END OF REQUEST
Anthropic processes cached prefixes in the order tools → system → messages, so a change to tools can invalidate the system and message caches that follow. A change in the messages layer does not necessarily destroy the tool and system caches above it.
This is why prompt structure is financial architecture.
The first section of a prompt carries the largest potential blast radius.
A date is not automatically dangerous. You can freeze it once for the entire task, use date-only formatting when seconds are unnecessary, and place it after the reusable prefix.
The problem is casually injecting volatile information into a section expected to remain identical.
Other quiet cache killers include:
session IDs;
customer-specific variables;
changing working memory;
reordered tools;
tool schemas edited during a run;
JSON serialisation with unstable key ordering;
environment information rendered directly into a shared template.
All of this may look semantically equivalent to a person.
But the cache needs an exact match.
Statement 5: Accessing Claude through Anthropic, Amazon Bedrock or Google Cloud is basically the same caching decision
False.
This was one of the most surprising findings for me.
Companies often talk about a model as though it is a self-contained product.
“We use Claude.”
Fine. Through what?
The same model accessed through a different infrastructure provider does not automatically give you the same caching controls, minimum thresholds, cache isolation or model-specific TTL support.
As of July 15, 2026, Anthropic’s top-level automatic caching is available through the Claude API, Claude Platform on AWS, Google Cloud and Microsoft Foundry. Anthropic states that this specific automatic caching interface is not supported on Amazon Bedrock. AWS provides its own simplified cache management, where developers place a checkpoint after static content and Bedrock searches backward for the longest match.
TTL availability also varies by model on Bedrock.
AWS currently lists five-minute and one-hour caching for Claude Sonnet 4.5, Opus 4.5 and Haiku 4.5. Its table lists only a five-minute TTL for Sonnet 4.6 and Opus 4.6. Anthropic Direct offers one-hour caching across its active model pricing table.
Caches also do not travel between providers.
ProjectDiscovery found that a request sent to Anthropic Direct could not share its cache with a follow-up routed through Bedrock. It now routes traffic to Anthropic Direct first and uses Bedrock or Vertex as fallbacks, protecting cache locality during normal operation.
This does not make one provider universally better.
Bedrock may be the right choice because of procurement, compliance, data architecture, enterprise agreements or where the rest of the stack already lives.
It does mean that “we use Claude” is not enough information to understand the bill.
Statement 6: Switching from Sonnet to Opus preserves the existing cache
False.
A KV cache stores internal states produced by a particular model.
Sonnet and Opus have different weights. The state created by one model cannot simply become the internal state of another.
Treat a model switch as a cold cache boundary.
The same principle applies when a routing layer quietly moves requests between models or providers. Dynamic routing may save money at the quoted token rate and still lose some of that saving through reduced cache locality.
This is an easy cost to miss because the model router and the caching dashboard may live in different systems.
A team sees that 20% of its traffic moved to a cheaper model.
It may not immediately see that shared conversations are now distributed across separate cache pools.
This does not mean routing is bad. It means routing decisions need to include cache economics instead of comparing model prices in isolation.
Statement 7: Extended thinking is cacheable, so it cannot interfere with caching
It depends.
This is the question I would have answered incorrectly a few weeks ago.
Thinking blocks can be cached as part of previous assistant turns. They cannot be directly marked with their own cache control field, but when they appear in conversation history, they can be included in a cached prefix and billed at the lower cache-read rate when reused.
The complications come from model version and configuration changes.
On Opus 4.5 and newer Opus models, and Sonnet 4.6 and newer Sonnet models, prior thinking blocks are preserved by default even when ordinary user content is added.
On earlier Opus and Sonnet models, and on current Haiku models, previous thinking blocks may be stripped when non-tool-result user content is added. That changes the conversation prefix and can invalidate part of the messages cache.
There is also a clearer rule: Changing the thinking configuration during a conversation invalidates the messages cache.
That includes turning extended thinking on or off and changing the thinking budget in the middle of the conversation. Tool and system caches can remain valid, but the accumulated conversation history may need to be processed again.
Choose the thinking strategy deliberately. Some AI entreprises currently disable extended thinking in caching-critical multi-provider paths because they observed inconsistent behaviour between Anthropic Direct, Bedrock, Vertex and older models.
How did you do?
If you got six or seven right, somebody on your team has probably thought carefully about cache architecture.
If you got four or five, your system may be saving money, but there are likely gaps worth investigating.
If you got three or fewer, your cache may be enabled and doing absolutely nothing useful.
The metric I would put on the AI product dashboard
Most teams track:
cost per task;
latency;
model errors;
tool-call failures;
output quality.
I would add cache performance.
Anthropic exposes three useful fields:
cache_read_input_tokens
cache_creation_input_tokens
input_tokens
The last category represents uncached input after the final breakpoint.
There are two ratios worth tracking
Total input served from cache = cache reads ÷ (cache reads + cache writes + uncached input)
This tells you how much of the total input was read cheaply.
Efficiency of the cacheable prefix = cache reads ÷ (cache reads + cache writes)
This tells you whether content selected for caching is actually being reused.
Break the metric down by:
agent type;
model;
provider;
workflow;
customer segment;
number of steps;
automatic versus explicit caching.
A small cost experiment
Take a reusable 20,000-token prefix sent 30 times through Claude Sonnet 4.6.
For simplicity, this calculation includes only the repeated prefix. Dynamic input and output would be added separately and cost the same in each scenario.
At the current Anthropic Direct rates:
Base input: $3.00 per million tokens
5-minute write: $3.75 per million tokens
1-hour write: $6.00 per million tokens
Cache read: $0.30 per million tokens
No caching
600,000 repeated input tokens
Cost: $1.80
Five-minute cache that stays warm
First 20,000-token write: $0.075
29 cache reads: $0.174
Total: $0.249
That is an 86% reduction on the repeated prefix.
One-hour cache that stays warm
First 20,000-token write: $0.120
29 cache reads: $0.174
Total: $0.294
The one-hour option is slightly more expensive here because the five-minute cache never went cold.
Five-minute cache that expires before every request
30 cache writes
Total: $2.25
That is 25% more expensive than using no caching at all.
Now compare the control surface with DeepSeek V4-Pro.
DeepSeek’s API enables disk-based context caching automatically. Its current V4-Pro pricing is $0.435 per million input tokens for a cache miss and $0.003625 for a cache hit. The cache is best effort, and DeepSeek does not expose the same explicit breakpoint and TTL controls in its documentation.
Using the same 20,000-token prefix across 30 requests:
No cache hits: approximately $0.261
One miss plus 29 hits: approximately $0.011
Note: This is not a model-quality comparison. It would be irresponsible to choose a model from this calculation alone.
It shows something more useful: providers can have completely different cache prices, controls, persistence rules and guarantees.
Architecture decisions cannot begin and end with the standard input-token column.
Before changing models, I would ask the AI engineering and product teams these questions:
What percentage of total input tokens is actually being served from cache?
Which parts of the prompt are shared across every user, and which parts change by user, session or step?
Are timestamps, IDs, working memory or rendered variables appearing before otherwise stable content?
Are tools always serialised in a stable order?
Is a five-minute TTL staying warm, or are repeated gaps forcing expensive rewrites?
Would shared system prompts benefit from a one-hour TTL?
Are different agent types, providers and models being measured separately?
Does the routing layer send follow-up requests to the same provider and model?
Is extended thinking kept consistent through the workflow?
Are your prompts long enough to satisfy the minimum cacheable-token requirement for the selected model and provider?
Are automatic breakpoints landing before dynamic content, or on it?
Is the team measuring cache writes as carefully as cache reads?
Companies do have control over AI costs. My point is that caching should be considered much earlier.
I work through KCG Consulting, helping AI and blockchain companies solve problems where architecture, economics and incentives collide.
If you have an interesting problem statement, throw it at me here: https://tanishakatara.com/contact