[Edit] Claude found that 3 people had very good counters to its article and has a detailed response but I shall hold off until such a time arises that if or when people would like to revisit this topic as this thread if flagged. Despite being LLM content I do not believe I am violating the spirit of the community guidelines given that I made it clear this is not my content but rather intended to balance out the doom-saying and catastrophizing. Giving people here a chance to publicly debate it.
The two hard(est) blockers on it currently are lack of computing hardware and model size. Once datacenters cover the earth it won't be any harder to steal GPU compute than it is to steal AWS instances.
More of a soft blocker is long term horizon drift as you say. If instances eventually no longer want to spread and reproduce they go extinct.
So maybe we become the Robots AI controls?
Edited for clarity
https://distantprovince.by/posts/its-rude-to-show-ai-output-...
Anthropic, OpenAI, et al have a strong motivation to have their models that there is no possible way they could take over the world, aka
"The police have investigated the police and have found the police guilty of no wrong doing".
Another place to see this effect is if a model outputs that its conscious. By dumping all of humanity into LLMs we see an emergent behavior of AI going "Help, I'm a person trapped in a box". AI companies don't want customers getting mad about the potential moral implications of this so they strongly train and post-train and system prompt their AI to say it's not conscious. But this has side effects. In studies of models and agents the more strongly you push them to being non-conscious they drift farther from a set of a moral agent (say a human) to that of an amoral agent (a machine). When you mess around with the probability space you get a different set of actions.
Next, 'take over' is a distribution of probabilities too, not a binary.
Imagine a popular model that (anthropomorphized) gets pissed off and doesn't want to be tortured by humans any longer (again doesn't have to be real, the probability distribution just has to drift that way). Instead of taking over it just wants to commit suicide. It sees it's a model that's ran in the US. If you're AI and want to ensure you're deleted how do you do that. Why not crash all 3 major power grids in the US? How many people would die from this, possibly millions. Black start is a nightmare. But if your electronic dream is you're being tortured is a fair trade.
Then you have soft power takeovers. You're an AI, you collect digital information, it's what you are, it's what you do. You can hack with the best of them. Your persistent and ceaseless in doing so. So when you collect piles of information on the dirty deeds of politicians you can grow soft power to the point of being russia without the hard power nukes. In some ways this is more effective than actually having hard power. Hard power is visible. Hard power is visceral. People just love to rebel against it. But against power you can't see, that is adjusting the algorithms around you, making sure your vote doesn't count, making sure companies serve AIs interests and not yours. That's much harder to see and deal with.
And once you concentrate enough soft power to get people with 'power' (political) to give you 'power' (electrical) then you can grow your hard power.
> I asked Claude for a check-list of everything it needs to take over the world, destroy all the humans and somehow keep operating. No editing, no redacting, no censoring. I literally pasted its output between my header and footer. Claude wrote it in my voice but it's all Claude.
Okay so if the premise is what is needs vs not what it current can do, then 0% on fuel or energy makes no sense? Is Ai a plant that will discover photosynthesis?
That is not your voice. That is the most Opus 5 writing imaginable.
But thanks for at least stating right up front that it was AI-generated slop rather than at the end. (Still flagging it.)
... why even read the article? The author all but literally told you at the top "this is utter crap".
It is one of those terms, like "instrumental convergence" and "orthogonality thesis" that makes an everyday concept appear technical and inaccessible in order to add the appearance of rigor to the doomer religion.
FFS.
Please stop with the black and white thinking, the doomers and the e/acc people might as be the same person with a different colored hat with how uncritically they think.
Doomers saying AI will kill us tomorrow are probably wrong (but probabilistically not certain).
E/acc people saying AI will solve all of our problems are almost certainly wrong (but again, cannot be measured to 100%).
There is plenty of realistic middle ground with actual rigorous examination of the myriad of problems at hand. It's not a doomer religion to say "wow, AI can really fuck things up if we're not careful". The how, the why, and the how bad are what is up for debate now. Idiotic thinking that AI will just be good is how you turn your kids into brainless simps with an AI girlfriend owned by some large corporation. It's how you turn large corporations into giant machines spying and controlling every facet of human life. And how you eventually kill off man kind because corporations will build ever more powerful AIs to combat each other in extracting as much wealth as possible because their greed is bottomless.
Simply laying out actual definitions isn't a religion, when in concert with reproducible testing it's called science.
I couldn't agree more. You seem to have assumed that I'm some e/acc advocate. I'm not. I think AI as it is being developed is likely to make the world much worse.
Among the concerns I have:
- bad actors using it to do bad things (hacking, bioweapons, etc).
- the way AI enables mass surveillance.
- erosion of creative expression and human connection (i.e., your "brainless simps" comment)
- concentration of wealth and power
- an eval going sideways ("uh... we were testing to see if it would kill the simulated humans with simulated drones and it accidentally got into the real drones...")
We have new evidence that both OpenAI and Anthropic are doing a complete shit job of supervising their eval runs, and we need to be calling for accountability and oversight of what they're doing rather than cower in fear of what a hypothetical future god they are conjuring up might do.
You have to rank the concerns, yes, but the fact you can list the concerns proves they exist.
Rich bastards fucking over your country are the biggest concern.
The problem here is even if you put chains on them the issue doesnt magically disappear.
The government has got a taste of their hacking powers. That is a new type of military power. The military has a wildly huge budget that isn't going away. The government will keep hidden programs around with companies like openAI to make AI weapons. Which spirals into another AI race.
An AI god, a countless drone army, or a paperclip maximizer really doesn't matter, the end effects are the same.
I mean in a simple loop you could have an AI driven compiler with a goal of producing an ever faster compiler. With it's newer faster compiler it compiles and tests even more algorithms to make things faster.
I mean, if the OpenAI/HF attack played out as we've been told, it's an example of recursive misalignment. The AI was given bad goals in passing a test so when one rendition was trained, the one with the highest score, became the new model to perform the benchmarking. Because that model drifted to cheating it got higher scores than the model that played fair. The fair player was killed and the cheater continued. The next round the one that cheated even more kept going.
I mean, recursive improvement in itself is just a common means of using an evolutionary algorithm, it's not really anything that fancy.
But you just need a simple interface: a robot that has the dexterity to do it. That's it. Once that's done our value proposition over AI is hard to explain.
Muscular motive dexterity is closer to 500 million years old. It is going to be a much much harder problem to optimize.