A grant limited to one selected file, with read and write access scoped separately and an expiry, would avoid giving the agent the whole Drive.
Could not find an easy way to do this. Connect agent (Muse) to Drive is opening up the whole thing. Please share steps if you can.