Prompt caching
Cached input tokens — where they come from and what they cost.
When a model recognises the beginning of your prompt from a recent request, it can reuse the work it already did on that part. Those tokens come back counted separately:
{
"usage": {
"prompt_tokens": 1204,
"prompt_tokens_details": { "cached_tokens": 1024 }
}
}Cached input tokens are priced separately from fresh ones — see Pricing.
The cache is the model's, not ours
Eterial does not cache prompts. It reports what the backend tells it and prices accordingly. Nothing is switched on or off from your side, and there is no cache to invalidate.
Getting hits
Caching works on a shared beginning, so the shape of your prompt decides whether you get any:
- Put the stable part first — system instructions, tool definitions, the long document — and the varying part last.
- Keep the stable prefix byte-identical between requests. A timestamp or a reordered JSON key at the top defeats it entirely.
- Send related requests close together. Caches are short-lived.
Why hits come and go
The same model can be served by different backends from one request to the next, and a cache does not follow you across that boundary. A run that was mostly cache hits yesterday can be mostly misses today without anything in your code changing.
Treat caching as a discount you sometimes get, not a cost you can plan around.