I feel the article gets it right when suggesting it's the amount of (not always accurate!) data they can throw at you. In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them. But if half of those examples are made up, well, that's a different story.
I still think of the commenter here who said "LLMs are a DDOS on free resources" and I feel the comparison works here. If police officers can overwhelm innocent people into confessing, then so can an LLM that "can't be bargained with, can't be reasoned with, doesn't feel pity, or remorse, or fear! And it absolutely will not stop, ever, until you are"... convinced.
A lot of internet arguments become about identity politics and supporting the right team, while dismissing any argument from the other side as presumed to have bad intentions. I’ve seen people argue online for things they didn’t really believe, but they didn’t want to give an inch to the other side. Taking up the counter argument is a moral responsibility.
As soon as the other side is revealed as a chatbot that my team versus your team thinking stops playing a role. For us in tech with an understanding of how an LLM reflects its training data and the intentions of its creators not so much, but for the people like the example in this article I imagine it causes them to let their guard down and be open to considering the other side. They can accept the argument without letting someone else win any points.
The particular problem with honest discussions is you are the only agent that you can be sure is having one. Honest discussion is formulated on trust and trust, as we are learning, is a very difficult thing to establish. For example, in my view anything involving advertising is likely a lie, or at least likely adversarial to my wishes. On the internet itself conversations are much more likely to drift into the adversarial too. Some of this could just be dialectic, but most often it's emotional investment by the other speaker. Also, even pre-AI the internet is a bullshit generation machine. We take all of our politics, advertising, and human stochastic parrots that are stuck on an infinitely running prompt then bundle up all this data and train AI on it, and wonder why AI acts like us.
We don't have tells for when AIs are lying to us, or when they're making stuff up.
When disagreeing with a human, it's very easy to view it as a competition. One is right, one is wrong - the one who is wrong is the loser. To change your mind is to be submissive to the other. I exaggerate, but I think we all feel this way at some point or another. It's why political arguments at Thanksgiving get heated. It's the fact that there's people who think something different, and think YOU'RE wrong - and vice versa! With a model, there's no person to get upset with, or to feel competitive with - to muscle for rank - or to temper your affection for while wanting to correct them.
The AI is only interacting because you asked, and clearly has no emotional stake in winning the argument. To change your mind in this context isn't to lose a contest. This makes it much more palatable to read rebuttals to your ideas - not to mention the tone and style seek to avoid offense to the reader as much as possible.
You're right. There are many many times when even then humblest hint that you are right will have negative interpersonal implications, which does make it hard to change minds.
I've also seen several times where I make a suggestion, humbly accept its rejection, and then, lo, a week later the other person has the same idea I suggested.
I'm used to machines malfunctioning, but having one willfully disobey me, and even with a touch of disrespect, is just...what a time to be alive.
The problem with AI is that it cant match human stupidity. It need some training on artificial stupidity to match its human counterparts. Humans on the other hand sit on a wide spectrum on the stupidity scale. Those of us binging on AI will become cognitively obese while those on an AI diet can flex their cognitive muscles.
It helps to think both are wrong and are just trying to figure out what right looks like, or what other information exists that was not considered when forming one’s opinions.
No it's not about humans being irrationally competitive. Human limitations on conversation length, bandwidth, research speed, etc are severe, creating a prisoner's dilemma around open-mindedness that usually makes it an unstable strategy. At any point, your conversation partner can choose to abuse the fact that confident lies take 1x effort to tell and 10x-100x effort to debunk -- unless you are both in a context that actually discourages this behavior, which is rare. Closed-mindedness is a Nash Equilibrium.
Instead, LLMs can be more persuasive due to economics. An LLM doesn't have to worry that it is wasting its resources trying to logic someone out of a position that they didn't logic themselves into, or worse, dumping the effort into a conversation with a bad-faith actor intent on exploiting the misinformation asymmetry. The resource allocation question was answered before it was even invoked, by the person paying to run it. The LLM is not playing a game where it will be punished for good-faith argumentation, so it can afford to do more of it.
I suppose that's a fair point as well. Though, if I'm arguing with a human - and they pull up ChatGPT to make their points and do their arguing for them, I would consider that bad-faith. Even if it might be the same exact dialog as if I pulled out my phone and discussed it with AI, without of the human middle-manning. Maybe I'm just particularly sensitive, but for me, there's something about my argument being with a real human that makes it much more emotionally charged, and prompts my mind to close. I'm aware of this and try to resist, but it's I think very natural
That whole episode caused the whole industry to shift away from RLHF, and towards RLAIF, RLVR, and DPO, and add a lot more safeguards, tests, and reward functions that push models in the direction of doing the opposite of what people want and confronting and strongly correcting their users, if it has determined the user is wrong.
1. It doesn't get tired or frustrated during a discussion.
2. It'll engage every single one of your questions/statements (besides hitting guardrails).
3. It'll appeal to the person's own ego even when the person is wrong and work around it.
4. It's not seen as a person (very important), but some of us or most of us at least in certain dialogues, end up anthropomorphising it. Think about that one time you thanked it for something, or when you got angry at it. This weird combination where we know it's not a person but irrationally we're still treating it as such in a way leads to a sort of disarming effect imo.
5. Many people see it as an authority figure in what is being discussed without questioning the results in many cases, even though we know that a. it was trained on human data and/or also searches up human data real time (and more worryingly other AI's data from news pieces, blogs... aka synthetic data that is also wrong), b. it gets things wrong all the time.
6. It's the perfect fence sitter depending on which version (guardrails) we're talking about.
Most of these can be replicated by humans who are good at understanding psychology and are just good talkers. The part you can't replicate is the sense that you're not talking to a person which lowers many barriers in people.
Also keep in mind that these can also be crippling weaknesses. For one being able to change a person's mind (when it works), can be used nefariously by the entities controlling the AIs training.
I've also unfortunately witnessed a lot of people who think AIs are somehow omniscient and/or omnipotent. Was very common on X and other social platforms with AI where people ask the AI questions it couldn't possibly answer because it made no sense for it to in the context at the time.
I see not everyone is familiar with how social network debates happen. It also seems they are quite convincing, as politicians are employing AI bots extensively.
I suppose I also grant it significant power to define me psychologically, and, in doing so, to understand what is happening to me.
My son wanted to wear his undies on the outside a la Captain Underpants and I tried war gaming this with him - hmm, so your undie on the outside is going to require another one inside your pants and we are out of fresh pairs etc etc. In the end, he gave up on the idea of mimicking Captain Underpants.
(I'm the dev)
I don't know which model underlies this. It responded more quickly than the models I normally use. It made the mistake I expected it to make, since this misconception is very common in training data, and conceded the point from my follow-up.
(EDIT: I guess that this link is not accessible to people without my cookies, except the developer I'm responding to above.)
Far too many screens to click through to get to the good parts.
And then, the good part starts and is immediately interrupted with some error. On retrying, o start getting a long answer which seems to make sense but o can’t read to completion because of another overlay which hides the bottom three-quarters of the response behind yet another nag screen about the research I’m supposedly consenting to.
I tried to make the site with minimal client side state, so I'd hope a refresh would fix it
Being confidently incorrect can still convince the masses.
Human intelligence is also powerful because when the assumptions we put into those shortcuts are incorrect things go terrifically wrong.
That’s not to say it’s the main factor, but I’d prefer that we remove the stigma we currently have around recognising anything to do with IQ.
https://www.researchgate.net/publication/325733310_The_Influ...
The remark is risible because it is an example of why it it false. Everyone is subject to influence by words, that's what words are for. Being "confidently incorrect" is also called being "mistaken", and everyone does it every day, including the commenter by the very act of saying it never happens.
It only takes a small modicum of thought to prove that the thesis is obviously completely false, yet it is stated with absolute sneering confidence and contempt.
By the commenter's logic, only someone with "low IQ" would believe their own words.
These are also not the only ways that people can be misled about things, and susceptibility to them is likely not exclusively correlated to intelligence, but they might well be heuristics that LLMs can leverage effectively.
Also that correlation I mention likely can be quantified, but I doubt anyone is doing the research on it.
I'm going to assume the actual truth behind this is complex, varied, and social.
When I was younger I had moved from a school that slightly above average to one that was below average in grades. The area had a really strong cult of ignorance feeling and I saw a lot of different behaviors in people that reflect themselves in political patterns. People that embrace ignorance seem much less likely to follow people that outwardly signal intelligence, quite often the opposite happens and they outright reject them. Instead of intelligence they are much more apt to believe those that broadcast success.
This quite often leads to power seeking individuals "playing dumb" and bolstering their success signal. "Awe shucks, that guy that got top scores at Harvard is just like me who didn't graduate high school" would be a good analogy to frame it.
You mean you don't independently know one way or the other?
So you are taking it on his authority?
Because it was stated with confidence?
According to him, you must be one of those low IQ people.
Does that help to see the irony here, and why the original claim is not only clearly wrong but inherently elitist and insulting towards its own intended audience?
The alternative, the world that all intelligent people must live in at all times accordinng to the comment, is epistemic solipsism, that everything is presumed false, everyone is lying or erroneous, because there is no guaranteed way a priori to separate true statements from false ones.
> I doubt anyone is doing the research on it.
Not only is there research on it, this is like Psych/Journalism/Philosophy/Law/Communication 101 stuff. There are textbooks on it written over a century ago. It is the topic that the field of "Social Psychology" was invented to study.
The dominant theory is the Elaboration Likelihood Model, which describes and predicts based on situations when a person is more likely to doubt a claimed statement of fact, taking into account past experience with the source, cultural norms, motivation, time, concentration/nondistraction, and domain knowledge.
There is also tons of research specifically on the impact of intelligence on susceptability to persuasion by authoritative tone, and the reality is the *opposite* of what was claimed: smarter people who enjoy deep thinking are more likely to believe falsehoods (Price & Stone, 2004), because they perceive the overconfidence as exciting new information to think about, just like you are doing now.
So congrats, by uncritically believing the guy who incorrectly said you were low IQ for believing him, you actually provided evidence that suggests you are high IQ, because science shows that gullibleness is a trait of high-IQ people.
You could also be right-wing authoritarian, that's the other thing that strongly predicts gullibleness but I'm gonna guess you're smart instead.
I'm guessing you didn't read what I wrote, or maybe I flubbed the communication in some ways because I was trying to be cooperative and complimentary towards you.
Edit: ok, you might object to me saying you're gullible, but the point I'm making is that that's not a bad thing and is actually evidence in favor of intelligence, the opposite of what the other guy was saying.
I'm gullible too. I'm also short, nearsighted, bad at communication, unable to bench over 90 lbs, and tend to spend way too much time researching fields irrelevant to my own work. Susceptibility to over-confidently incorrect messaging is a scientifically proven achilles heel of smart people. It's important for us to know that so that we can metacognitively account for it and hopefully avoid falling victim to it by way of scams, cults, and hallucinating AIs.
More likely in my opinion it's the context that matters here. If people know they're trying to be convinced of something and know they're talking to an AI, they may feel like they can trust (what they consider to be) an unbiased and rational AI. There's probably a fallacy associated with the assumption that an AI convincer is more rational and fact based, while it's more probable that the AI in the real world is, in fact, funded and trained by a think tank that wants you to vote against your own self-interests.
I suspect that LLMs not being human allows people to not just anthropomorphise the robot, but they project themselves upon it. When someone speaks to the LLM, they're kind of, or actually just literally talking to themselves. But then something new! 'Themselves' suggests new information to themselves. And now without the scary / icky meat and blood human on the other side, said person accepts the argument on its own merits.
Of course, the concern now is arguments based on false or statistically hacked data.
At least to me this is a given.
You can take the same set of factual information and give it to one person that studders, has poor presentation, and otherwise poor vocal cadence and people are going to have a hard time with it.
Now, if you took the same facts, maybe even the exact same paragraphs and gave it to Richard Feynman, even if you didn't know who he was, the presentation itself is likely to hook you.
Anecdotally I've noticed a trend of far-right bots/trolls moving on from just spouting hatred and really lean into cherry picked and misrepresented statistics to give themselves an aura of credibility. That strategy seems moderately effective because the argument consists of verifiable facts even though it forms a faulty conclusion.
So all of this is to say that this is a double edged sword where malicious actors will have the advantage.
It sounds like the arguments were basically AI slop hallucinations.
Hell no. More choice is not an infinite money glitch.
Putting more and more and more options to users is how you overwhelm systems, till people simply perform the default, least challenging action as a reflex.
That is the current state of the information ecosystem, it is controlled by overwhelming consumers, not by controlling content.
Ask a Democrat or Republican to sit down and ask a chatbot, something it will answer contrary to.
And yes, both teams are wrong about things.
Do you firmly believe they will change their mind? Or will they claim the stats are wrong, or that the AI leans one way?
Facts (2+2), don't need a mind change. Ideas which are grey, abstract, are not going to be changed, and all research indicates that political mindset is almost indelible.
The movie "Don't Look Up" was a comedy built upon this truth.
I wonder if a properly prompted LLM, or if a very intelligent LLM could actually do that?
Seriously though, you've mentioned an ongoing human fear, machines deciding what you think.
There are all kinds of machines that tell us what to think. I would consider any system that abstracts away the human to be a machine in this case. Society itself is one of these machines.
Language is possibly one of the most important things people can have a working knowledge of, especially now that there is so much of it. When you send a prompt to an LLM you're telling it what to think. When it sends text back, its telling you what to think, but you're at a disadvantage, when you think it changes you. The LLM outside of its context is read only.
We put 2 + 2 in our calculator, we get 4. We've spent decades pushing computers as accurate. Making them accurate. We trust the machine and the process to give us the right answers when we put in the right data.
So when we disagree with the computer, we doubt ourselves.
I rarely disagree with my calculator, but I often ask LLMs to explain why they did something, and I sometimes need to correct them, adjust their assumptions and nudge the goals they stated. These things are a lot more like us than my trusty TI-59 (still going strong, BTW).
> Even in Hackenburg’s recent preprint, Claude spouted numerous inaccuracies and falsehoods
> when Hackenburg ran his competition of coached elite debaters and AI, there was one way he could bring AI down to human levels of persuasiveness: by forcing it to write human-length messages at human writing speed.
> they found that an AI could talk people into conspiracy theories, and that the magnitude of their increase in belief was roughly the same as that of the decrease in belief after talking to a debunking bot.
None of this seems good.
The biggest problem with using LLMs is that it prevents you from going _outside_ the distribution. It always flattens. It can be fixed but that's how it works today.
As an example, take something that the world converged on today that is incorrect and ChatGPT will agree with it. In a few years when society changes, chatgpt changes along with it. It doesn't do first principles analysis.