Was super confused about the leak vector cause I was thinking in HTML terms not javascript. Found a nice paper that illustrates the timing attack

https://link.springer.com/content/pdf/10.1007/978-3-319-1846...

Having read that I'm surprised they haven't done a hybrid approach. store the site key with the cache but also the initial download time. If a site requests a resource that's cached but not for that site don't go download it but artificially delay it's delivery from the cache such that it's indistinguishable from a normal fetch (utilizing the original time maybe with some variance) then log the new site key such that further requests from that site appear to be using an isolated cache.

while you don't get any real first-time performance boost you do save actual bits over the wire so there is real life impact and savings while eliminating the source of the timing attack.

Google is deciding who are the winners on the web that get better performance (and surprise, google is chief among them, such as the youtube player) - with a very weak justification about why this penalty for security/privacy is worth it (basically "we allow 3PC tracking already, so this can't be worse than that")