LLM APIs price input and output tokens separately, and long conversations inflate input costs as they grow. Prompt caching stores and reuses prior computation, cutting model serving costs by up to 90% and returning cached responses in milliseconds instead of seconds. Cached replies are identical to fresh ones for identical inputs, which matters for enterprise reliability.
Two caching strategies dominate production systems. Provider-side prefix caching stores the key-value cache for repeated prompt prefixes such as system prompts or few-shot examples. When a new request shares that prefix, the provider skips recomputation, reducing input token costs by 50–90% and time-to-first-token by 10–85%, depending on the provider and prefix length. Application-level semantic caching stores full API responses keyed by embedding similarity and returns a cached answer for semantically equivalent queries without making an API call. Research suggests 31% of LLM queries are semantically similar to earlier requests, so the second approach has real headroom.
Vendor features differ. Anthropic reports up to 90% cost reduction and 85% latency reduction on long prompts, with cache reads priced at $0.30 per million tokens versus $3.00 per million for fresh processing. OpenAI offers automatic caching with a 50% discount on cached tokens. The two strategies are complementary, not alternatives: semantic caching yields 30–80% cost reduction on hit rate, while prefix caching yields 50–90% on prefix tokens. A production system can use both.
Prompt structure determines whether prefix caches get hit. Place static content, including system instructions, few-shot examples, and document context, at the start of the prompt. Keep dynamic parts like user questions at the end. Shared prefixes must be byte-identical across requests; injecting timestamps, user IDs, or other variable data before the static block invalidates the cache.
Keep tool definitions and formatting instructions stable across requests so they remain part of the reusable prefix. For semantic caching, normalize the dynamic tail so equivalent questions produce similar embeddings and are more likely to pass the similarity threshold. The same discipline applies to both layers: separate what changes from what does not, then make the static part as large and stable as possible.
A multi-tier architecture captures the different optimization opportunities. Check an exact-match cache first, using a hashed prompt in Redis. Then check a semantic cache, using embeddings with a similarity threshold. Only on a miss call the LLM API. This three-layer hierarchy is the standard pattern for production systems.
Self-hosted caching with Redis or pgvector gives you control over cache keys, data residency, and eviction, but adds operational complexity. Choose it when you need semantic caching or cross-provider portability. Provider-side prefix caching requires no extra infrastructure but ties you to vendor-specific behavior.
Caching earns its complexity mainly for high-volume, low-variability workloads. If prompts are highly dynamic and cache hit rates stay low, the added infrastructure may not pay off. Measure hit rates before committing to a design.
Track cache hit rates, cost per request, and latency percentiles. Adjust the semantic similarity threshold based on your workload's tolerance for approximate matches. Version system prompts and invalidate cached entries when prompts or tool definitions change. Treat user-specific context as part of the cache key so one user's data is never served to another.
Avoid caching sensitive or tenant-specific responses in shared semantic caches, or scope cache keys by user or tenant and apply access controls. Evaluate output quality on representative queries because semantic caching returns approximate answers. Monitor for drift when prompts or underlying models change. The same caching discipline that cuts costs also exposes the failure modes: stale answers, cross-tenant leakage, and silent quality degradation. Measure them explicitly.
