Sometimes, I take breaks in the middle of a session and end up losing the prompt cache which drives up the token usage a ton. Don't all the providers have different cache times and behavior? If one person takes 20 minutes between messages, some services will keep that cache while some won't. Is there a way to factor that in?
Rough current state: OpenAI's automatic caching typically keeps prefixes warm for ~5–60 min depending on load; Anthropic's default cache TTL is 5 min (extendable to 1h for an extra write fee); Gemini's explicit caches let you set your own TTL with per-hour storage pricing. So your 20-minute coffee break is usually fine on OpenAI, a cold cache on default Anthropic, and whatever you paid for on Gemini.
Adding an "average gap between messages" input that derates the hit rate per provider is a really good idea — putting it on the list. Thanks!
I can’t decide which is more disappointing.