> Because AI agents are stateful, they frequently pause for seconds or hours while waiting on a database lookup, an external software tool, or human feedback. To make our accelerators more power-efficient during those pauses, our new Agent Compute Engine software automatically suspends idle agent sessions and shifts working memory off the chip in milliseconds, cutting operational infrastructure costs
Nice. I wonder how they manage to move kv cache from working memory to disk so quickly to enable speed ups when pauses are a few seconds at most