Prompt caching
Reduce cost and latency by reusing prompt prefixes, and control which requests share a cache.
This page explains how Engy caches prompt prefixes and how you can improve your cache hit rate. It is intended for developers who send long or repeated prompts, such as chat applications and agents.
How caching works
When a request starts with a prefix that a server has already processed, the server does not process that prefix again. Engy bills those tokens at the cached input rate. For the rates, see Models and pricing.
Caching is automatic. You do not need to turn it on or to mark cache breakpoints.
The number of cached tokens is reported in the response:
| Endpoint | Field |
|---|---|
/v1/chat/completions, /v1/completions | usage.prompt_tokens_details.cached_tokens |
/v1/messages | usage.cache_read_input_tokens |
Engy does not charge for writing to the cache. On /v1/messages, cache_creation_input_tokens is always 0, and cache_control markers are ignored.
Each server holds only the prefixes that it has processed. Your cache hit rate therefore depends on which server receives each request.
Default behavior
Engy sends all turns of a conversation to the same server, so that the conversation history stays cached. Engy identifies a conversation by its system prompt and its first user message.
This applies to multi-turn requests and to single prompts of about 2,000 tokens or more. Shorter single prompts are distributed across servers, because they are inexpensive to process again.
Set a cache key
To decide which requests share a server, send prompt_cache_key on /v1/chat/completions. Requests with the same key go to the same server, regardless of their content.
curl https://api.engy.ai/v1/chat/completions \
-H "Authorization: Bearer $ENGY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "kimi-k3",
"prompt_cache_key": "support-bot-v3",
"messages": [
{"role": "system", "content": "You are a support assistant for Example Corp."},
{"role": "user", "content": "How do I reset my password?"}
]
}'If you do not send prompt_cache_key, Engy uses the user field in the same way.
When the server for a key is busy, Engy sends the request to a fixed second server for that key instead of waiting. The second server then also caches the prefix.
The /v1/messages and /v1/responses endpoints do not accept a cache key. The default behavior applies to them.
Share a prefix across users
If many users share a long system prompt, a separate key for each user spreads that prompt across many servers, and each server processes it without a cache.
To improve the hit rate:
- Derive the key from the shared prefix, so that requests with the same prefix reach the same servers.
- To spread a heavily used prefix across several servers, append a bucket number to the key. For example, combine a hash of the prefix with one of 16 buckets.
- For multi-turn conversations, derive the bucket from the conversation, so that all of its turns reach the same server.
Keep prefixes identical
The cache matches exact content. Place the shared content at the start of the prompt, and keep it identical between requests. A value that changes, such as a timestamp or a user name, starts a new prefix at that point.