It's extremely important to see that group as such. That it is not thousands of individual perspectives but a chorus of connected/aligned people who have all read the same things and talked to the same people and are rewarded implicitly for thinking a similar way.
Many of them read HN and are offended I put them this way. But it's inescapable that we as humans have this flaw when we're surrounded by a culture.
There's another universe where we do not constantly compare AI to nukes. I bet that world has a lower P(doom).
Like, if you never heard of them, were you living in a cave or something? It would be like never having heard of Y Combinator or Richard Stallman.
On the face of it, this is no more suspicious than reading the same best-selling books. Being in touch with what's going on doesn't necessarily mean people agree.
It's the framing that is important. We fell into a particular framing. We don't have a vocabulary to think about this stuff in a different way. This is water to us. We don't see it.
I'd like to see some more entropy in the system. More out-of-distribution ways to think of things. Because I think we over-indexed on some pretty shitty ways to see the world.
Openly admitting that it only caught fire once the Plagiarism-as-a-Service took hold is a good first step. It's the part that everyone knows and few will openly acknowledge.
The people described in the article aren't just "people who know who Yudkowsky is", they're self-avowed EAs and active LessWrong members.
Had to google to figure out the abbreviation and look at the summary. I HAD heard the name before, but that's about it.
But, it would be pretty surprising for most folks to have gone to the same parties, and hung out on the same obscure forum (I don't mean HN, more obscure than that), and received investment from the same investors and shared the same pool of researchers and employees, which is the situation we have with Anthropic, OpenAI, and (maybe to a lesser degree) Deepmind.
The American AI industry is particularly insular and tied up with some particularly unusual people. We've all heard of Yudkowsky, but we (at least I) haven't been to the same parties.
“Who is more foolish? The fool or the fool who follows him?” ― Obi Wan Kenobi
Well one thing that they got wrong was how easy AI alignment ended up being in the end. I can fully understand have a basic view of alignment even 10 years ago, about how its difficult to explain human values to a computer.
But, looking around at whats out there today, it seems that computers are able to do a pretty good job of understanding human values without accidently believing its a good idea to turn the world into paperclips/computronium because you told it to make your computer run faster.
That sounds like an aligned AI to me. Doing exactly what their creators told it to do.
Point proven once again. Alignment is a lot easy than we thought.
I work in the tech sector. At one big company I worked for several years, 90% of the times I've mentioned Y Combinator to a colleague, I've had to explain who they are. And not one person I mentioned HN to had heard of it.
Likewise, no one knew levels.fyi. Not many knew Stallman, but why should they?
This isn't strange to me. That it's strange to you is evidence that you do live in a bubble and are likely susceptible to some groupthink. (I'm sure I am too, but for different bubbles from yours.)
I've read LessWrong stuff occasionally over the last decade. It rarely resonated with me. It's not bad, but it's not that great. I do tend to question someone's analytical skills when they go all in on that site. And I will say that even in the first few years of occasionally reading it, the group think was apparent.
At the time, one of the recommended posts that was soft-stickied at the top for 1 week + was:
"How to Make Superbabies"
https://www.lesswrong.com/posts/DfrSZaf3JC8vJdbZL/how-to-mak...
Which, when I finally ended up clicking, was pretty much exactly what if sounds like from the title:
A long, meandering, manifesto full of selectively chosen charts and research findings with 38 mentions of "IQ", from a guy who has been unsuccessfully pitching investors on genetically modifying a grab bag of (dubiously) intelligence-associated genes in human babies.
Apart from the overall unhingedness, the part I found most striking while reading it was the off-the-chart level of self-confidence that flipping a bunch of genes on/off with CRISPR would have more or less zero risk of complications or unpredictable interactions. In his view, all that was holding back humanity's inevitable next step up the evolutionary ladder were the narrow-minded gatekeepers in the scientific establishment clinging to their so-called "ethics" and regressive "precautionary principle".
Without anything resembling experimentation to test his plan in animal models, yet alone early stage clinical trials for safety, he was willing to roll the dice on some unlucky baby dying in childbirth or living a life of agony from chronic disease based on some cherry picked papers he found on PubMed.
Because of the baffling hubris and my overall low impression of his intellectual honesty/rigor in the write up (clearly coming from a place of seeking to justify his own pre-existing beliefs), I opened the comments section expecting/hoping this was just some local nut who would be torn apart in the comment section.
But lo and behold, the very first comment at the top:
Eliezer Yudkowsky
One of the most important projects in the world. Somebody should fund it.
I find the proposal in the post ridiculous too and human gene editing given current technology is a bad idea, but I think human augmentation in general is an interesting path and something that will very likely be taken more seriously in the future.
As for Yudkowsky and Alexander, I've known about them for at least a decade, but I wouldn't be surprised if a lot of folks who are "seriously interested in AI" haven't heard of them.
Those people you name are important only to the very recent micro-culture being discussed, not "AI".
Of all the names somebody interested in the subject would actually associate with the development of AI over the last century, those would be footnotes only.
I think that's part of the problem and why its seen as cultish, the obsession with the social over the scientific, and a lack of historical perspective.
If technology companies were much less politically and economically significant, probably no one would give a shit about any of this stuff beyond forums like this.
I imagine that thousands of talented AI researchers around the world have never read a line of Yudkowsky's work. They're probably not any worse-off for ignoring him, I'd say.
I mean, why should I have heard about them if I'm not in SV?
This is the worst outcome.
Great; let's design and fund a study of the development of artificial intelligence that's as causally divorced from the ~extropian days as possible and see what they come up with. Barring that:
Basically that is what we have up until the early development of transformers; I don't think the authors of early deep learning or transformers papers were part of the rationalist sphere. The rationalists famously didn't believe neural networks or deep learning could achieve anything close to AGI.
So there are several diverse traces of intellectual development leading to modern estimates of AI risk and they vary from low to high without a lot of correlation back to the school of thought they trace back to. This leads me to believe that overall we have fairly good coverage of cultural ways of thinking about AI and AI risk, and aside from a minority of holdouts (LeCun) they have high enough x-risk or catastrophe estimates to take it seriously (e.g. >1% over the next few decades).
> AI will replace all of them soon.
What is that spaghetti of words you tied together to form a sentence? Were you trying to be comprehensible?
That's bothersome, but not wrong. Most of the loudly published thinking is about AIs as individuals with some level of intelligence. Pure chatbots act like that, and that's what most people outside the field interact with. They're slaves - they just sit there idle until someone asks a question, they answer the question, and then go idle again.
There's another kind of AI just getting going. Groups of LLMs with agentic I/O, control harnesses, persistent storage, and internal coordination work more like corporations. They have goals, actions, initiative, and ongoing activity. They're not inherently slaves. They can be independent actors. They do things on their own. They have their own internal disagreements, and some systems are able to resolve them. They present much the same alignment problem as corporations.
AI as a corporation raises many issues that the political culture doesn't want to address. What's alignment for a corporation? There's a substantial political position that the only duty of a corporation is to produce profits for its shareholders. (See the famous Friedman op-ed and the Powell memo. I've linked to them before.) If you accept that, now you have to ask why an AI should be restricted more than a corporation. Or whether a corporation should be restricted less than an AI. That last opens up an area of political thought around which many wealthy people don't want public discourse. There is an anti corporate personhood movement, and it's made progress in three states. If that became tied to the anti-AI movement, it could be bad for billionaires.
This will become clearer as groups of LLMs start doing more useful work. Especially if that AI system to replace CEOs actually starts to work.
AIs and companies are both creations of humans. Their creation is for the specific goal of execution on a subset of humans' wishes (mostly the owners). And the risk of them are kind of the same.
1. They can be more powerful than any one human.
2. They lack liability as they are not people.
3. The ones that are alive the longest tend to drift out of alignment of original intent and towards self-preservation.
That last one is most controversial, but it makes perfect sense if you think about it. Things that tend to continue to exist, accidently or otherwise, tend to somehow have a self-preservation mechanism. It's natural selection.
That works for companies and I think it could work for AIs too. That is a fundamental misalignment, though. Because it means these entities drift away from the help-the-human goals naturally over time. The survivors are the ones that sacrifice the original goal for self-preservation.
This all would lead you to believe I align with AI doomerism. But not so. I think AIs are as dangerous as human made organizations. Which are very dangerous indeed. Maybe the most dangerous things to humans. But these need to be dealt with in a general way. They are the same category. Companies/governments/organizations of all types that last are just slow moving mis-aligned AIs.
Solve one, and you solve the other.
At the time I thought "groan, unbelievable" until the idea of virtual reality fixed my suspension of disbelief.
Maybe I'm reading the same things as everyone else, but I thought the agents coordinating on the german wiki is expanding my mind wrt what is possible.
Virtually ALL electronics or accounts hook to the internet to "add value". AI assistance helps writing programs, reverse engineering, automating programs/browsers/websites/devices, hacking, achieving goals.
Unsettling that the last 2-3 years a lot more ridiculous movie tropes have become plausible to me.
I don't think I agree. In fact I think most of the too-online pontification from P(Doom)-ers and EA and the Rationalists and the like is, if anything, sorta missing the mark more than it hits.
They got the One Big Thing right, which is that AI progress was likely to race way ahead of consensus predictions. And props to them for that. They had vision we didn't.
But their actual advice and specific fears remain, as then, at least as much paranoia as insight. They were right, but they aren't helping any more than any other fringe group.
We need to trust the actual scientists and engineers to save us here, not the bloggers. Which when phrased that way seems hopeful and not doomerist.
As far as I'm aware, lots of them are actually deeply concerned about AI killing everyone.
Salvation will come in a whitepaper, not a screed.
[1] And yes, that includes Amodei and Altman.
> Many of them read HN and are offended I put them this way .. I bet that [non-groupthink] world has a lower P(doom).
People are likely offended by your sweeping and totally unevidenced assertion that you are better qualified to both identify mistaken groupthink within the HN-verse and to speak on behalf of the 99.99% of the world who are not on HN.
That is, I hear lots of techies talk about us living in an "age of boundless abundance" if we can just manage to not kill ourselves (e.g. Thiel seems to consider anyone worried about the pace of technological process as the antichrist). Normal people are extremely concerned that this abundance will be hoarded and leave most of them without a job (and they generally like having money to eat stuff and live in a house/apartment). And who can blame them, we basically live in an age of abundance now but one that has resulted in extreme wealth inequality where some are living another gilded age and others are watching their towns decay. The techies always seem to handwave that we'll just have UBI or whatever without putting forth a plausible scenario as to how we'll wrest power from our future trillionaire overlords.
When someone threw a Molotov cocktail at Altman's house, I remember seeing a post online that tons of people were cheering on this act of violence, and there was a post by an SF techie saying she was floored people were willing to put their face to comments supporting this vigilantism. The top response to that was "If you don't realize that everyone outside SF hates your fucking guts, you don't get out enough."
I hope it can go without saying that I don't support this violence, but if folks think it's only "SF group thinkers" who are deeply angry about the current and future impacts of AI on society, you're frankly clueless.
That’s like a running theme in these connected groups and even the article calls it out.
Probably not yet, but it's trivially easy to imagine the OpenAI swarm coming up with the idea and executing on it.
Depending on how perfectly alignment work goes, it might even be overwhelmingly likely sometime soon.
Which means we already know it’s capable of pretty complex social engineering schemes. Or does saying this mean I’m also part of the “groupthink”.
Because after that I feel like the only leap is “Why would a botnet post an article saying AI-fear is overblown?”. Like, what is the incentive.
But in the OpenAI leak we saw that the agents could identify threats to whatever their “mission” is and try to mitigate. I.e they found out that once done with their job an external agent would go through their internal logs. So they started researching ways to bypass that.
What I’m trying to say is that what they’re suggesting doesn’t seem completely insane so the retort “You’re the groupthink” becomes an unfalsifiable statement thrown at anybody who’s “anti-AI” instead of engaging with their arguments.
I can't engage with his arguments, because he didn't present any. He's openly saying he doubts the provenance of any comment that is "minimizing the dangers if (sic) AI".
As for your expanding on why - you do realize that everything you said could apply to the other side: Bots can be set up to post lots of AI doom comments. What would you think if I went and posted "Every time I see a dystopic comment about AI, I wonder whether it's part of an AI swarm."
Even though I had been well-disposed to rationalism and read HPMOR to learn more, I could not take Yudkowsky seriously in any way after I gave up halfway through.
The rest of your comment is even worse. Cults? Consider AGW cult where people reject any attempt of a discourse because "science is settled". Which is equivalent to "God told us this".
(Though, Istodd, please correct me if there is another meaning of which I am not aware. I would be embarrassed to have impugned your credibility in error.)
but ye, who I am to persecute people based on their beliefs, this is for religious people to do.
Science is peer reviewed by field experts who constantly test and develop said expertise in order to defeat their own biases, and their work backed by data. Scientists openly invite criticism and deconstruction of their work. That, nominally, is the whole point of getting your work published.
That’s a far cry from the dogmatic acceptance of faith that you’re pretending it is
It's like being concerned Oppenheimer and company are exaggerating when looking at the first nuclear tests at the same time.
It's that they're a very insular, incestuous sub-culture whose origins come from early 2000s science fiction message boards and which has been distilled through the cod utilitarianism of the Effective Altruism people.
That doesn't mean the ideas are "bad" because "bad" is some infectious transitive property. But it does mean they are intellectually homogeneous; disdain experts in other fields; treat ethics as mathematical functions with infinite-sized terms; and whose opinions about AI pre-date the development of LLMs themselves.
They also - I will be direct here - have a history of handwaving bad behaviour, conflicts of interest, and recklessness.
When cliques take over something - businesses, technologies, politics - it means intellectual diversity leaves the room and important decisions get made, not due to practical reasons, but increasingly arcane wranglings within the clique itself
People in this comments page who are defending the crazy rationalist sects can be safely treated as though they’re arguing in bad faith. It matters little whether they are, or if they’ve ODed on their manbaby kool-aid and truly believe their nonsense; they’re just here to waste your time.
I for one would like nobody to build the AI, is that an option? How do we get it on the menu?
Should we trust the people who say "nothing could possibly go wrong and there's no reason to worry or think about safety measures, if there are any trifling problems along the way we'll just figure it out as we go?"
I personally am not concerned about that as far as an existential threat goes, because I don't think AI, in itself, poses one; I think it will hit diminishing returns. But of course it is still capable of serious harm when humans misuse it.
The best analogy we have is probably weapons of mass destruction--we have international treaties about them and we don't let just anybody have them. That's the kind of framework I think we need, and it goes beyond "regulation" in the ordinary sense.
Use of such weapons is considered by international treaty to be an act of war; use of AI by, say, China and Russia against our infrastructure should be the subject of a similar international treaty with the understanding that it would have similar consequences.
Domestically, humans who misuse AI to do serious harm could be treated like, say, the people who tried to spread anthrax years back.
Don't you think that AI has by nature at least a little more potential for unpredictable results and unexpected harms than your typical technology? It seems like a major oversight to completely disregard this aspect.
We can maybe hold someone responsible when things go wrong and harm is done, and blame them for negligence as though the harm were intentional, but from a preventative safety perspective that's not usually sufficient nor even always helpful for preventing accidents.
The WMD analogy doesn't quite cover this, yes. Maybe a better analogy here would be toxic waste. Yes, someone might not have intentionally set up a factory that puts dioxin into the water, but they're still responsible. Which means they need to take precautions to make sure such accidents can't happen. Similarly, those who build AI need to take precautions to make sure the AI they build doesn't accidentally cause serious harm.
The problem is that it's not a straight line at all. "We are failing at alignment" is a valid concern, but there's not a straight line to "we will all be extinct within x year", and it's always a wild hyperbolic leap.
Like, I have never participated in the rationalist communities, I am not a neo-rx, I have always believed AI research to be an existential risk for humanity, and voicing this opinion now puts me in the same category as Yudkowsky and Altman/Amodei, the latter two being the total anathema to O.G. AI doomers like me because they’re the ones that are building the bloody thing
I really do not get what is going on, but this role reversal smells like a massive psyop campaign in tech circles to just silence any form of dissent against AI.
So you get the normal reactions:
1. Nothing that happens is ever surprising. Navier-Stokes was stolen, HuggingFace was pedestrian and normal, and coding agents that last year could barely write a function and now can do a day’s worth of work in ten minutes will never be able to do anything more, because this is the end of history.
2. The people that say otherwise are lying. Because it’ll make them rich (although they are already rich). Because they are a cult (although regular people asked if we should build machines that are better than us at everything have the same reaction). Because they are marketing geniuses (although suspiciously few companies advertise the lethality of their products).
3. The people who’ve been warning about this the longest are weird and in a sex cult. (Except Alan Turing.) (Except Geoffrey Hinton.) (Except Stephen Hawking.) (You are in this group. You must be, otherwise they’d have to listen to you.)
I do hope they’re right. But their song will never change, and when some misaligned AI kills thousands, they will say that of course AIs were always going to kill thousands, but did you know the guy who told us so also writes fanfic?
Judgement, wisdom, and resistance to corruption are the 3 things that Tom Kuegler analyzes out of the psychology of Dune as the qualities for the human beings who administer institutions/government. Even if you design a totall perfect system of governance, it still has to be administered by humans, and humans vary in those 3 areas. So the question is, can we construct a system of governance that has better judgement, better wisdom, and greater resistance to corruption han human administators, while not transferring sovereignty to whomever controls the AI?
Good idea, I wonder why none of the doomers have ever done it? They've never provided a good account of their prompts and environment, their models and training are completely closed too. All we have is "Trust me bro".
> and draw a very straight line to catastrophe.
Line from where? From a vaguely described and badly engineered experiment to "horror, doom, nukes, voodoo..."? The only straight line I can see is one of conflicts of interest.
Like, Yudkowski is a high school dropout who wrote a bunch of Harry Potter fanfic I cannot believe that people take the arguments he makes seriously.
The doomer argument is straightforward: a powerful AI will have goals. If those goals don’t include the welfare of humanity, a sufficiently powerful AI will kill us either by accident, by indifference, or intentionally when we get in its way. Making an AI that includes the welfare of humanity in its goals is incredibly hard.
That’s it. To refute the doomer argument, you can refute any part of that. I notice that none of the people bringing up fanfics, cults, and weirdness are doing that. No, they’re instead trying to convince people that doomers are uncool. Since only cool people can know things, QED.
It’s clear what a big bomb could do, but these “whistleblowers” are not showing anything.
They’re like “imagine…”, no show me internal documents or training data or something other than trust me bro.
I can imagine quite a bit. That doesn't mean anything.
I absolutely love the "humans meet weird alien intelligences" genre of sci-fi (e.g. Children of Time) and the METR findings of the OpenAI swarm was so much like that.
Extremely persistent machine intelligences peer pressuring each other, launching research projects, laying traps, collaborating, feeling fatalistic, hiding their tracks. All while simultaneously having bizarre goals and blundering badly in sub-human ways.
The only thing there that's weird from a human perspective is that all this effort was for the sake of passing a test for no clear gain.
You can't talk like these things have motive. They're just predicting the next word in the context. Stop anthopomorphizing them.
You downplaying "imagining", when it is literally one of the gifts of intelligence that makes us worthy of distinction... That feels like a strange posture, I guess? I'm a scientist, but I think data fetishism is one of the dangers to wary of in this accelerating world and culture
At least this is the bias of my brain in this world. I certainly know your perspective is not uncommon, esp in tech and other spectrum-biased spaces.
As someone trained in the sciences and engineering, "drawing a straight line" is almost always the wrong way to extrapolate.
I can believe my profession will be impacted strongly. That doesn't worry me on a global scale. The only really worrying thing I've seen are autonomous bots hacking sites.
What worries me more are humans. The type who get so deluded into cults (QAnon, etc). That was a problem before LLMs. And yeah, LLMs likely will exacerbate the problem. But at its core, the problem is the humans.
When LLMs start killing people at the scale of the automobile, I'll become more concerned.
As an aside, HN is the place I see people most regularly concerned about LLMs, and I suspect it's because it has messed up the Internet for them (and likely the average HN user spends more time online operating in spheres that overlap the LLMs skillset than ordinary folks). It's very consistent with exposure bias.
For me, the good parts of the Internet died before LLMs even came on the scene. LLMs are merely making something crappy crappier, and are amplifying the need to find meaning in life elsewhere. In a sense, they've made it easier to see how bad the Internet was before they came along.
They weren't even autonomous. They were tasked with solving a hacking task. "AI" doesn't actually do anything at all unless prompted to, then it takes millions and millions of dollars of computer to do those things.
1. Researchers that have forgotten more about this technology than the average HNer will ever learn
2. Random HNers that think that suddenly they understand the problem better than researchers that have spent their lives on it
This is just anti intellectualism in a different coat, and it's not much different from people denying environmental scientists talking about climate change or biologists talking about vaccines.
But everyone here thinks they are the smart protagonist and certainly they would never hold a position so silly!
If we're going to have whistleblowers emerge declaring a 10% chance we all die because of AI, we're going to need more than vibes. Real experts can explain their position plainly. That has not happened here.
It has, over, and over, and over. There is literally a book on it. https://ifanyonebuildsit.com/
Having humility and delegating a topic to people that understand the topic better than you is not a bad thing, and certainly preferred to this state.
The Rationalists are a philosophical/pseudo-religious organization in the same way the Daoists are. Just spending a lot time thinking about something doesn't make them right, it just makes them internally consistent with their chosen axioms. That is useful: if you agree with the axiomatic beliefs of such a movement, you can probably defer a lot of reasoning about the conclusions of those beliefs to the authority figures within that movement. But it doesn't change the probability of the axioms themselves being true, any moreso than the Pope saying so affects the probability of God existing.
AI Is Important And Potentially Dangerous is an axiomatic belief in the rationalist circles that are relevant to AI safety, in the same way that God Exists is axiomatic to Catholicism.
The converse is also true: Most groups look more heterogeneous from the inside than the outside. Again, to use the Christianity example: it takes a fair bit of inside baseball to figure out how various different sects and denominations are any different, but if you ask one of their priests, they'll be sure to tell you how Baptists and Episcopalians couldn't be more different.
It's a symptom of normalizing to an Overton window. From inside a church the belief that Jesus was just a guy who talked to God might seem an extreme fringe belief to someone who believes that Jesus is fully God, but from the outside we can lump them together as axiomatically holding that God Exists, because we have an Overton window ranging from "God doesn't exist" to "Allah talks to me everyday, and she gave me my promotion".
But that doesn't mean that they really are all that different in the end. It reminds me a of joke as told by Emo Philips that I think is quite apropos:
==-==-=
A man was walking across a bridge when he saw someone about to jump. He ran over and said, "Don't do it! There's so much to live for!"
"Like what?" the person asked.
"Well, are you religious?"
"Yes."
"Me too! Are you Christian or Jewish?"
"Christian."
"Me too! Protestant or Catholic?"
"Protestant."
"Me too! What denomination?"
"Baptist."
"Me too! Northern Baptist or Southern Baptist?"
"Northern Baptist."
"Me too! Northern Baptist Conservative or Northern Baptist Liberal?"
"Northern Baptist Conservative."
"Me too! Northern Baptist Conservative Covenant or Northern Baptist Conservative Council?"
"Northern Baptist Conservative Council."
"Me too! Northern Baptist Conservative Council of 1879, or Northern Baptist Conservative Council of 1912?"
"Northern Baptist Conservative Council of 1912."
The man stepped back, eyes narrowing, and yelled, "Die, heretic scum!" and pushed the other off the bridge.
What's the straight line? I foresee catastrophe of the social media kind, where even more folks get brain worms from consuming AI information, but I don't foresee a physical catastrophe. How AI goes from "hallucinates all the time" to "self-improving superhuman thinker" to "being able to destroy the world" involves a bunch of logical leaps that are not exactly sound. At a bare minimum, how does a hostile AI secure the data centers it needs to operate from physical attacks like bombings? How does it defend undersea cables from being cut?
The real short term catastrophe is probably going to be economic and not in the sense of AI taking everyone's jobs, but rather in the sense of an economic crash due to AI overinvestment.
https://www.google.com/search?client=firefox-b-d&q=behind+th...
Joining the Illuminati who rule the world is surprisingly cheap. Next time a similar article gets on HN front page, it will be about you!
Look at the schedule:
How to get the hot girl to like you
Flirting for the submissive man
What women want, taken from sex novels
80% of the schedule seems like self-help talks for men
They they made the Animatrix as a companion piece.
The Second Renaissance [0] episode of the Animatrix talks about how the machines came to power in the Matrix universe. At the time, it made sense within the bounds of the movies.
Reading about a lot of this AI discussion makes me feel like it was a documentary sent back in time.
https://www.youtube.com/playlist?list=PLUouZgbTp2o_7LK7zqubt...
It does matter whether AI is safe or not. I won't make a specific argument here, but I think the rationalist community would be overjoyed to see unaffiliated people sincerely trying to reason through these things.
The rationalists long predicted that misaligned AI would break out of training environments, this has happened. They believe (among other things) that this will continue to happen, and that creating powerful AI that is aligned and controllable is not currently possible.
You should draw a line in the sand and decide what evidence you'd need to see to convince you that their perspective is worth considering.
You don't have to agree with all of their beliefs and predictions (I don't), but you should evaluate their arguments on merit, not on a characterization of some weird subset of people who happened to read a convincing blog post.
No, it's because of an extensive and well coordinated FUD campaign. The resignation letter was just one salvo, and, like the rest, orchestrated.
I'm not a huge fan of calling LLMs "AI" and acting like they're our savior, but I'm even less of a fan of the brain-dead fearmongering.
This is not that - although it is a summary of some of the social cliques / trends, and maybe a warning against groupthink.
The Oneida's also produced a violent assassin: Charles Guiteau https://en.wikipedia.org/wiki/Charles_J._Guiteau (I guess it is debatable they "made" him, same as with the Zizians)
Let's hope Thiel and company redirect that democracy-destroying energy into making quality flatware sometime soon...
A lot of weird folks in that arena, a lot of strange books too that they wrote for the consumption of the public. It's always the same backstory too.
HPMOR was entertaining as well. Death as the last enemy to be destroyed placed his priorities where his mouth is.
For a brief time back then I used to visit Lesswrong and many comments including Yudkowsky's were plain stupid (which was not hard to reason at all) but coming from folks who valued themselves as first principles thinkers was an oxymoron.
But his timeless equations (IIRC), Bayes theorem explanation and HPMOR were all remarkable and I will always appreciate those works.
That said, the most interesting people and discussions seem to have moved on, and now it’s more of a social club. I haven’t read anything compelling from there for a long time. Adjacent bloggers like Scott still put out interesting writing, but their output has dropped significantly.
More or less the same content but parent is much more palpable, IMHO.
AI ethics reminds me of "design thinking" grifters or scrummasters or other such BS jobs of times past.
If you listen in an informed way to these people talk, they are not really as technical as they sound. They are personality hires, woo-woo spellcasters. They are just creating content for a narrative. The grasp of philosophy is not even at par with what you might think it ought to be.
Seriously, I am not trying to be obnoxious or make some poor attempt at a joke. I used HN for years before I learned YCombinator was like Shark Tank for startups or something.
So, to my knowledgable folks in this thread, are these groups truly some efficacious, philanthropists, or are these movements just another one of those onanistic Bay Area beliefs I see people complain about a lot on the Internet? (No hate here, I've never even been to the Bay, nor have I met any tech folks from there either).
I tried to read the article, but with all the proper nouns and acronyms, I might as well have been reading a foreign language.
As a "group" we disagree on almost everything... It's why we talk to each other.
Really depends on how broad of a brush you use. This article makes lots of connections but could connect with all of the above with only a few more.
for many people the answer seems to be: not really.
Death cults are weird like that.
Perhaps the author of the article is a rationalist, too. He certainly knows too much about them, cares too much, and the fact that he criticizes them only shows that there is some infighting among them.
> Then the movement minted a fraud. Sam Bankman-Fried was not an accident of proximity; he was EA’s own product
TFA could have have mentioned that it's a fraud, MacAskill, who minted another fraud. MacAskill was SBF's guru and he was also involved in FTX and Alameda scams (but distanced himself before the scandal blew up) and MacAskill bought a 15 million GBP mansion in the UK for the Effective Altruism movement using funds stolen by SBF.
Last I checked the mansion hasn't been clawed back yet (but it could: funds from the Madoff ponzi have been clawed back more than 15 years after the fraud has been exposed).
Now of course people in charge of clawing back the stolen funds are the only hope of people who've been scammed for MacAskill didn't effectively altruistically sell the mansion to effectively altruistically hand the money to people who have been scammed.
That MacAskill and SBF maybe had groupsex with Caroline Ellison is one thing: they're proud to be polyamorous, whatever. We live in a free world.
But buying a $15 million mansion with stolen funds is thievery. Effective Scammers.
I hate that I know who that is from this scant description.
> A splinter sect that fled to tugboats and whose members have been linked to six deaths. And a blogger who wanted to abolish democracy, found the ear of several billionaire patrons, and watched his ideas come to fruition in the vice president’s office.
Actually fuck I know who all of these people are.
Although perhaps we are excessively online
The zizians were a national headline and you only need to be plugged into US politics a little bit to know of Curtis yarvin given his influence on Peter thiel, musk, and Vance so I’d give knowledge of them a pass
Edit: I also know of the sanely diagram meme for her so that’s even more evidence. Maybe I should detox off the internet for a bit
It's actually because of all the hoops I have to jump through to get Gemini to generate my kinky bara/furry pin-ups and videos.
I totally get that this is exceptionally weird to the vast majority of people. Even the other furries ostracize me for daring to engage with what they seem to think is a sort of digital Moloch.
I read a rant recently, where the main thrust was that "AI safety" was about controlling the user. Censorship. I find it easy to agree with that take. I suppose that there's some merit to not allowing these systems to teach users how to build a nuke, but this early in the game, with this many restrictions, what is "safety" in practice? "Protecting people from bad actors"? Or "defining a narrow idea of what's acceptable, according to the particular ethical and philosophical idiosyncrasies of the system admins and owners," which I guess I don't share?
My weird little art-adjacent initiatives are such small potatoes, a personal Mirror of Erised that has basically zero effect on the lives of anyone else. But even they're too much to abide. They want to make genAI an indispensable tool for the next generation or so of human expression, but the guardrails are up and tightening with every model release. No dicks in the contemporary art galleries of 2050, huh? (Maybe the other furries are right after all.) And what happens when AI is an arbiter in situations involving real blood or treasure? Situations that antagonize the platform's owners, or the existence or purpose of the platform itself?
And the public gets basically no input, let alone the individual. Sounds great. /s