I would not, most legal departments would not, all CSO's and compliance officers would not if someone explained it to them honestly. I have no doubt some will be tricked into approving such a thing and will try to back-peddle when it backfires on them.

Would you ever try something like this?

No I would not but I have only worked for companies with highly sensitive data, financial data, credit card data, proprietary code and data.

What safeguards would you absolutely require?

The entire AI stack would need to be written and maintained by the same company it is running in and all of the data must be stored in that companies data-centers. The interface must be behind multi-factor authentication and a corporate VPN running in the data-center. It would need to be audited by internal auditors, red team pen testers, external 3rd party code and infrastructure pen-testers and would have to go through the strictest change control. Every action by the AI must be highly audited real time and every action must be predictable and reproducible. No third party connections whatsoever. Any attempts to connect outbound must trigger and immediate mandatory all hands on deck response. The entire stack both client, agent and servers must run entirely within the data-center and not someones laptop regardless of how locked down their workstation or laptop is.

And that is even before factoring risks such as hallucinations, confidently accepting its own incorrect decisions. Blaming the AI for downtime, leaking customer data, leaking intellectual property would not be acceptable.

Having said all that, I am certain there will be some interested that could get it approved. Some companies give Okta root access via an agent to all their server fleets with no local guardrails. Should they ever get hacked that is insta-root on a lot of servers. My opinions on that matter are not suitable for public forums.