Everyone wants to make their scientific data archive work with Agents and AI, but this requires a complete rethink of the storage layer. Luckily, the recommendations end up being very similar to what data providers should have been doing anyway: forget about files!
This post takes you from "what is a chunk" all the way to understanding why there is really only one sensible way to manage large scientific datasets in the age of AI.
It's also a choose-your-own-adventure post - I didn't want to have to write the same thing 5 times for 5 different fields of science, so I just made one post in which you can toggle examples between different fields!