The magical super-AI will turn into a basilisk and start torturing every human that didn't assist in building it by simulating them forever.

Or something like that.

The doomers unironically panicked when Roko wrote that (vaguely based on their nonsensical "timeless decision theory", I think as a joke) and banned discussion of it: https://en.wikipedia.org/wiki/Roko's_basilisk