Prompt Caching

Simple Definition

Prompt caching is a feature where an AI provider saves the processed version of content you send repeatedly, like a long system prompt, a document, or instructions, so that the next time you include the same content, the AI doesn’t have to reprocess it from scratch.

The result: faster responses and lower costs on repeated calls.

Why Prompt Caching Exists

Every time you send a request to an AI model, it processes every token in your input, including parts that never change. If your application always sends a 5,000-word document as context before every user question, you’re paying to process those 5,000 words every single time, even though they’re identical across all requests.

Prompt caching stores the processed state of that repeated content. On the next request, the AI picks up from where the cache left off, skipping the repeated computation.

Where Prompt Caching Helps Most

Use case	Benefit
Chatbot with a long system prompt	Saves reprocessing the system prompt on every message
AI over a large document	Don’t reprocess the document for each question
Multi-turn conversations	Cache early messages; only process new ones
RAG systems with fixed context	Cache the retrieved documents across similar queries

Cost Impact

Anthropic (for Claude) and OpenAI offer prompt caching at reduced token prices, often 50–90% cheaper for cache hits. For high-volume applications, this can cut API costs significantly. Response times also drop noticeably when a large input is cached.

Limitations

Cache entries expire after a period of time (often a few minutes to hours)
Not all content is eligible for caching, only the portions explicitly marked or automatically detected as stable
Cache hits are never guaranteed, the provider decides when it’s beneficial