A power lifter who puts 400 lbs over his head is not a person we call strong only in retrospect. It’s not a dice roll that determines whether the slim girl or the muscled man can do that.
In modern times, the chances to "do what you want" and make society do what you want, are already very different for different people based on their generational wealth, family connections, ties to powerful men and power structures, etc.
A tiny minority coming from nowhere might be admitted to this exclusive club, and be celebrated as "meritocracy" wins. But actual power goes to people who inherited it from powerful people or to people propped up and favored by powerful people.
Every other country. We’re in a new age of realpolitik. If e.g. Cuba decided to start running on AI, we’d likely deem it an illegitimate government.
I do not want vibe-coding the law, especially criminal law. I do not want vibe-coding the tax rules. I do not want vibe-coding traffic safety.
And, in fact, we won't be governed by AI, even if we are. If we're governed by AI, we're really governed by whoever trained the AI, and/or whoever curated the training data. Do we want to be governed by them? Again, no, with expletives.
If someone assumes AI will become significantly more capable than humans at reasoning through complexity, then I can empathize with their opinion. I was previously convinced (open to) this possibility, but in recent years and the better AI gets the clearer it is to me that it's going to take a lot longer, and the super AGI outcome is a lot harder to see.
I'm sure by the time it could possibly be a feasible and positive option people will be plenty ready for it... So no need to prepare prematurely.
TLDR: I agree with you, but without the expletives.
That is, AIs don't get to govern things without being put into governmental power by humans. Don't let the humans be the voters.
That doesn't stop office holders from using them like a magic 8-ball. So step 2 is, vote out office holders that let AIs write laws (or even replies to constituents).
We're not going to be governed by AI because we're ready, it's going to be because the people who own it have secured enough power to make it happen. Whether it takes the form of government or capitalism.
Of course it will be every bit as bad as the people who implement it. But that just kinda highlights the core problem.