Every conversation contains miscommunications, a good communicator is just better at identifying miscommunication earlier, and knows how to remediate.
The mathematicians will always be needed to refine the goals (iterating a wrongly stated puzzle until hitting the sought after solution). Mathematicians will be those who understand the miscommunication between the human and the machine. Non-mathematicians will just see gibberish either way.
The threat is to the status quo: what happens the first time modern cryptography is broken? lots of damage and a lot of drama and discussions in order to "prevent it from happening again".
Then it will happen again. Again lots of drama and this time how to "prevent it from causing so much drama everytime it inevitably happens" (in hindsight this position will turn out to have been expressed in the first round of drama, and then promptly ignored.
Every time it happens the bootloaders and firmwares can be signed by anyone. Open source enthousiasts will write 100% FOSS bootloaders and firmware which they can finally self-sign.
This will disempower nation states around the globe (surveillance state goes blind when it comes to consumer electronics, harder and harder to penetrate the trustzone properly secured.) Surveillance temporarily becomes expensive again (requires actual manpower and bribing people to snitch and manipulate each other Stasi style).
It takes a while until COTS hardware with new hardcoded crypto for the bootloader has replaced the liberated consumer electronics which slowly start failing and need replacement.
At a loss on how to design new cryptosystems at sustained rates, AI will be consulted. AI can just as well be consulted to design cryptography with a backdoor (which it can break itself, but humans didn't know yet).
Different nation states and power blocks run their own AI to generate (backdoored) cryptography hoping diametrical power blocks or their citizens naively adopt it. Backdoored and unbackdoored cryptography become indiscernable to mathematicians. Subliminal communication or other forms of implicit collaboration may cause AI of different power blocks to conspire against nation states.
This results in a race to advance cryptography faster than anyone else. Simple heat dissipation alone voids Earth-based mathematics for finding mathematical breaks cryptography: what is not yet economic or cost efficient today will be in the future, so its a race to bite the bullet and start preparing and doing it now. This is why you see all the talk about putting AI datacenters in space.
It is not secure to shed such amounts of heat on Earth, even if you had the energy to start with. (heat death suicide does not lie on the principal variation)
It is also not secure not to participate in the cryptographic math race. (loss of cryptographic independence from foreign entities is also not on the principal variation)
Hence the computations go to space.
Those who consider space based AI datacenters nonsense are either not comprehending the looming mathematical cryptography race, or pretending they don't (in order not to wake adversaries).
The first rockets and satellites were also not profitable, government can sponsor technologies with reasonable probabilities of success in order to accelerate progress on a front.
Yes it is currently not economic to send AI datacenters to space. But since it will be the race to space based datacenters is on.