A
COST
Read the same input for less.
Representative example: cache-hit input costs 10% of regular input
FIELD GUIDE · 2026.08
USING AIISSUE 07
AI CACHE HITS
THE COMPLETE GUIDE
Even with the same model, cost and speed can vary dramatically.
The key is knowing how to freeze the beginning of each request.
Understand it in 30 seconds 8 min read
REQUEST / 0042LIVE
01Tool definitionsFIXED
02System instructionsFIXED
03Reference documentsFIXED
CACHE BOUNDARY
04Current questionCHANGED
05Latest resultsCHANGED
Keep the beginning. Add only new information at the end.
CACHE READ PRICEREPRESENTATIVE 90%↓*
MAX TTFT GAIN IN STUDY31%↓†
CORE PRINCIPLEFIXED PREFIX, CHANGING TAIL
* Representative official GPT-5.6, Claude and Gemini pricing · † Maximum in an arXiv study of 500 agent sessions; some strategies showed no gain or were slower
01
HOW IT WORKS / THE MECHANISM
AI doesn't remember the answer;
it reuses the computation for the same prefix.
Prompt and context caching does not remember the answer itself. During prefill, the provider reuses computation for the matching prefix. If the cache remains valid, only the newly added tail must be read.
01Prefill
Reads the full input and builds its computation.
02Cache reuse
Skips recomputing the same prefix.
03Decode
Generates the response one token at a time.
KEY POINT
Caching mainly reduces input prefill. Output-token cost and generation speed are separate.
INTERACTIVE LAB 01
Break the cache yourself.
Where the change occurs determines how much can be reused.
SIMULATIONGood flow: 3/5
01Tool definitionsHIT
02System instructionsHIT
03Reference documentsHIT
04Current questionNEW
05Latest resultsNEW
The first three blocks are reused unchanged.
With a typical prefix cache, less content after the change can be reused.
02
IMPACT / WHY IT MATTERS
Cost, speed, throughput. One hit changes all three.
Cache benefits vary by provider and model. The 90% below is a representative input-price condition, not total-request savings.
90%↓
B
SPEED
Skip long prefill work and get the first token sooner.
Results vary with prefix length and infrastructure
C
THROUGHPUT
Do more work with the same budget and time.
For most models, Claude excludes cache-read tokens from ITPM
INTERACTIVE LAB 02
Hit rate tells only half the story.
This simple model prices cache reads at 10% of regular input. Write, storage (TTL), and output costs are excluded.
RESULT / PER TURN
56K
Effective input load per turn, normalized to full-price tokens
Input cost index28 / 100
Estimated savings72%
98% HIT900K × 0.118 = 106.2K
VS
70% HIT50K × 0.37 = 18.5K
Even with a higher hit rate, an oversized context is about 5.7× heavier.
CASE
OUR ORCA USAGE RECORDS
ORCA HQ · 2026.07.20—07.27 · LOCAL USAGE LOG
Our own records show the patterns that bring the cache to life.
We analyzed 207 locally aggregated HQ sessions whose paths could be verified. Token counts are sums of provider usage fields; costs were not estimated without billing records.
- Observation periodMEASURED
- 8 days July 20–27
- Orca sessionsMEASURED
- 207 HQ paths verified
- Recorded responsesMEASURED
- 12,554 with usage
- Models usedMEASURED
- 8 Claude, GLM, DeepSeek
WHAT THE LOG SHOWS
Share of all input-related tokens accounted for by cache reads
Of 2.201B total tokens, 2.161B were cache read, 38.82M cache write, and 799K fresh input.
MEASUREDcache read ÷ (read + write + fresh input)
Cache readMEASURED
2.161B98.20%Cache writeMEASURED
38.82M1.76%Fresh inputMEASURED
799K0.04%Average per turn172K cache-read tokens
high reuse, but a large absolute context
THREE REAL RUNS
In hour-long concentrated runs, cache reads appeared across hundreds of responses.
All three are Orca benchmarks using Claude Opus 5. Totals are rounded.
01 · Operations document reviewOPSREV3
615turns
- Run time
- 57 min
- Cache read
- 137.1M
- Cache write
- 298K
- Read ratio
- 99.78%
Observation: reads were about 460× larger than writes.
02 · Scenario reviewSCENAREV
486turns
- Run time
- 49 min
- Cache read
- 110.2M
- Cache write
- 292K
- Read ratio
- 99.73%
Observation: most input-related tokens were reused.
03 · Long-form summary reviewLONGSUM4
425turns
- Run time
- 48 min
- Cache read
- 100.5M
- Cache write
- 264K
- Read ratio
- 99.74%
Observation: long-form work also showed high reuse.
WHAT WE CONFIRMED
All three runs continued work with the same objective within one session over a short period.
Caution: this is a strong correlation pattern, not a direct classification of each cache write's cause.03
PRACTICE / THE PLAYBOOK
Freeze the beginning. Change only the tail.
Cache optimization depends on how you start, continue, and clean up a session.
Decide what stays fixed before you begin.
- Organize long-lived rules and references
- Finalize model, MCP, and tools
- One task per thread
Add only what changed while working.
- Current question and latest results at the end
- Move large images and logs into files
- Continue the same task when possible
When it gets heavy, clean it up.
- Compress context for the same task
- New thread when the task changes
- Check hit rate and absolute tokens
General chatbot / PRESCRIPTION
One thread per topic; materials once
- 01
Organize the project description and materials once.
- 02
Continue the same task in the same thread.
- 03
Describe only what changed in long documents.
CACHE & CONTEXT HABITS
These habits hurt reuse and context efficiency.
- ×Dynamic values in system instructions
- ×Frequently changing tool lists or order
- ×Editing and repasting the same document
- ×Letting images and raw logs pile up
- ×Mixing unrelated tasks in one thread
WHAT TO MEASURE
Track ratios and absolute amounts.
01Cache read / write
Are hits actually occurring?
02Total input per turn
Is the session too heavy?
03Cost per task
How much does each result cost?
04TTFT
Does the first token arrive sooner?
Your workload's baseline and trend matter more than a universal ideal hit rate.
04
GPT'S TAKE / A MODEL'S TAKE
Cache optimizationisn't a prompt trick;it's information architecture.
Hit rate is useful, but 100% is not the goal. Reuse stable facts and reread only what is needed now.
OPTIMIZE IN THIS ORDER
- 01Remove unnecessary context
- 02Fix stable information
- 03Separate threads by task
- 04Measure hit rate
If caching conflicts with freshness or accuracy, choose accuracy first.
05
30-SECOND PRE-SESSION CHECK
YOUR NEXT SESSION
Check just six things before you begin.
0 / 6
Every check makes the session lighter.
ONE LINE TO REMEMBER
Good caching =
prefix stability × context-size control × work rhythm
How were the numbers calculated?
Estimated savings = hit rate × (1 − cache-read price / regular input price). At a 10% read price and 80% hit rate, input cost falls by about 72%.
Does caching work the same for every AI?
No. TTL, minimum length, discounts, and usage fields vary by provider and model.
Does staying in one chat guarantee a hit?
No. Provider policy, TTL, and internal context management can change the result.
Primary fact-check sources
Official docs and the original study were checked as of August 13, 2026. Orca totals were recomputed from local usage logs.