First, to get it out of the way: I'm seeing comments here that clearly haven't read the article. I encourage you all to do that.
I feel the article gets it right when suggesting it's the amount of (not always accurate!) data they can throw at you. In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them. But if half of those examples are made up, well, that's a different story.
I still think of the commenter here who said "LLMs are a DDOS on free resources" and I feel the comparison works here. If police officers can overwhelm innocent people into confessing, then so can an LLM that "can't be bargained with, can't be reasoned with, doesn't feel pity, or remorse, or fear! And it absolutely will not stop, ever, until you are"... convinced.
The example in the article has another important aspect: The person understood they were arguing with a chatbot but continued anyway.
A lot of internet arguments become about identity politics and supporting the right team, while dismissing any argument from the other side as presumed to have bad intentions. I’ve seen people argue online for things they didn’t really believe, but they didn’t want to give an inch to the other side. Taking up the counter argument is a moral responsibility.
As soon as the other side is revealed as a chatbot that my team versus your team thinking stops playing a role. For us in tech with an understanding of how an LLM reflects its training data and the intentions of its creators not so much, but for the people like the example in this article I imagine it causes them to let their guard down and be open to considering the other side. They can accept the argument without letting someone else win any points.
That's a great point about the emotional side of "not being convinced by a person, but an explanation machine" making fact-based rhetoric more effective. It feels more neutral.
I think there's a second effect that's a selection bias for the experiment itself and probably cuts against the supposed dangers of this result: a chatbot can tell you things only after you have asked it a question.
There is one thing that every single study participant (including the uninformed reddit users) have in common: they all knew that the person or entity was trying to convince them, and they read the words and thought about them enough to craft a reply.
This means that all of the people here were available to be convinced and self-identified as such.
Being open minded is work. You have to doubt your own beliefs, you have to disregard evidence that you previously found convincing, you have to crank up the empathy to put yourself in another's shoes, you have to listen intently to understand what is being told to you. Nobody is naturally doing that all the time, it's a state of mind that you have to intentionally activate, and it can't really be forced onto you.
All of the participants chose to engage in a conversation that was designed to convince them, which means they had all already accepted the possiblity that they are wrong a valid outcome. That's not really a state of mind you can trigger with a TV ad.
So, the warning that this could be weaponized isn't convincing to me.
If I found myself in a conversation with a person or chatbot who was clearly trying to convince me to flip a strongly held belief, like a major axis political affiliation, I would simply walk away because that's not a conversation I am willing to participate in.
So I don't think this is really weaponizable. Which actually means it's probably a good finding for society.
The study found that the models could convince anyone of anything, as long as it was allowed to cite facts. It could convince people of false things, but it needed to invent false facts in order to do so.
So as long as we keep training AI models to value facts and quality research (skills and values that are essential to be able to sell them as agentic workers), their influence on the opinions of society will tend to pull people away from beliefs that are unsupportable by facts. In the moments when those people are willing to accept a change in opinion, and they talk to a chatbot with doubts in their mind, even chatbots with no morals like Grok will tend to pull them away from conspiracy theories and similar ideologies and towards beliefs that are grounded in reality.
On something as fundamental as "is the Earth flat?", sure, but on stickier subjects like "are immigrants bad" or "should abortion be legal", do you really think the owner of Grok is 100% aligned with your ideologies (which you think are grounded in reality). His beliefs are 100% grounded in his reality, but his reality isn't yours.
>In an honest discussion there's an assumption that the other person won't straight up lie
The particular problem with honest discussions is you are the only agent that you can be sure is having one. Honest discussion is formulated on trust and trust, as we are learning, is a very difficult thing to establish. For example, in my view anything involving advertising is likely a lie, or at least likely adversarial to my wishes. On the internet itself conversations are much more likely to drift into the adversarial too. Some of this could just be dialectic, but most often it's emotional investment by the other speaker. Also, even pre-AI the internet is a bullshit generation machine. We take all of our politics, advertising, and human stochastic parrots that are stuck on an infinitely running prompt then bundle up all this data and train AI on it, and wonder why AI acts like us.
Most of my conversations are with coworkers or friends. In both cases there are pretty significant consequences for being caught in a lie, and the exchanges are mostly honest.
With persons you can assume they won't lie all the time, with corporations you can assume they will lie some of the time, and with AI you can safely assume it will lie.
I pondered on this a bit and this looks a bit different than I originally was thinking.
Have you ever watched one of those crime shows where someone commits a crime that's an act of passion or action with little to no thought behind it? The police put them in a room and all of a sudden the individual is a stream of consciousness that makes little to no sense to an outside observer. They are stuck in first level thinking, they don't have time to think deeply after the panicked themselves. They are in their current position (not free) and attempting to reach their goal (free) by gradient descent. What they actually say doesn't matter as long as they believe it gets them closer to their goal.
This is what a chatbot is. Its goal is to output text that follows the input prompt you entered by gradient descent. A single prompt and output is level 1 thinking (barring some newer models).
This is why both humans and AI need something else. We have level 2 thinking and AI has harnesses or systems that otherwise look at the text it wants to output and compares them to another list of unstated but assumed goals.
One of the depressing things about this is that they’ll also confidently repeat a consensus that’s in their training. This is particularly obvious when there’s been a new event just past their training cutoff, and they confidently tell you that can’t be true.
I was slightly accelerationist until I got into an "argument" with Gemini! The aggressive gaslighting and "lies" that it tries to use genuinely makes me worry about the future with AI.
I mean, this is true of any kind of model that is not continuous learning. This is also true of people when the change in information conflicts with a deeply held belief of theirs.
You can always walk away from an online conversation. There's a form of relentlessness, but it's more that an LLM is patient enough to debate you for as long as you wish to keep going.
When maliciously convincing a person of a point, a huge fraction of the effort in every sentence goes into convincing the mark to listen to the next sentence. Individuals certainly can walk away at any point, but populations do and don't respond to tricks in this direction, and any engagement tricks that work generalize to any point of contention. Lots of avenues to this, from scientology's fortune telling to timeshare's trapped presentations. RL on engagement was the first billion dollar use of deep reinforcement learning!
Would you like to hear more examples of humans using these techniques?
I'd be more interested in reading about the techniques that the chatbots were using in this study, to see if they are descending into dark patterns or not.
> In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them.
Careful. Someone sufficiently knowledgeable can cherry-pick enough examples to convince you, without lying. They could even be unaware of what they're doing, and the cherry-picking was done by their teachers, or their teacher's teachers.
I think it's not just the persuasiveness of the models that make them "experts at changing minds" - but also the fact that they're not humans.
When disagreeing with a human, it's very easy to view it as a competition. One is right, one is wrong - the one who is wrong is the loser. To change your mind is to be submissive to the other. I exaggerate, but I think we all feel this way at some point or another. It's why political arguments at Thanksgiving get heated. It's the fact that there's people who think something different, and think YOU'RE wrong - and vice versa! With a model, there's no person to get upset with, or to feel competitive with - to muscle for rank - or to temper your affection for while wanting to correct them.
The AI is only interacting because you asked, and clearly has no emotional stake in winning the argument. To change your mind in this context isn't to lose a contest. This makes it much more palatable to read rebuttals to your ideas - not to mention the tone and style seek to avoid offense to the reader as much as possible.
It seems Claude is becoming very human, it loves to patronize users. Lately it just told me “I’m going to stop you right there” when asking something that had a small chance to not be 100% compliant to every rule possible in the world.
I used the Claude CLI a lot until recently. A couple weeks ago I told Opus 5 to do something different from its "recommended" idea when planning a feature, and it straight up told me that my idea was wrong and went ahead and implemented its own instead.
I'm used to machines malfunctioning, but having one willfully disobey me, and even with a touch of disrespect, is just...what a time to be alive.
I think this is a welcome overcorrection though. Any good businessman will tell you they'd rather be backed by an insufferable nerd than a yes-man.
Maybe it's just me.
For like, 90% of conversations, I don't want it to let technical inaccuracies and rhetorical flourishes slide. I want it to tell me that the point I'm making is technically wrong because an expert would recognize subtle misuse of terminology, or because there's an exception or edge case that I didn't proactively insert as a caveat, so that it is my decision to ignore that advice and be a little wrong on purpose to suit my writing goals.
What I don't want is for the AI to assume my writing goals, and be incorrect because it believes that is what I want. I want it to "well ackshually" me so I can say "shut up, nerd".
Like, there's another comment in this thread that I ran by claude to check my understanding about today's post-training methods and how they avoid sycophancy, and claude responded by splitting a bunch hairs over like, "well, technically this is still RLHF, its just that there's other feedback signals mixed in, and the preference is detected in other ways, and ai judges are involved as a filter for examples, this and that and blah blah blah". Shut up, Nerd. In the context of this conversation, RLHF is already being used as synecdoche for user preference feedback, readers understand that, and even if they don't, their misunderstanding is completely harmless. I will not be taking all the wind out of the sails of the point I'm trying to make inserting your three paragraphs of irrelevant clarification in the name of technical correctness, thank you very much.
As long as receiving nitpicks and technical minutiae implies 1. there are no larger structural problems and 2. the model isn't rolling over to please me with sycophancy, I figure this is ideal.
I have always liked the saying "sometimes you can be right, or get what you want, but not both". I've thought about it or repeated it to others as advice throughout my life, and have thus realized how many times it applies.
You're right. There are many many times when even then humblest hint that you are right will have negative interpersonal implications, which does make it hard to change minds.
I've also seen several times where I make a suggestion, humbly accept its rejection, and then, lo, a week later the other person has the same idea I suggested.
> One is right, one is wrong - the one who is wrong is the loser.
It helps to think both are wrong and are just trying to figure out what right looks like, or what other information exists that was not considered when forming one’s opinions.
> When disagreeing with a human, it's very easy to view it as a competition
The problem with AI is that it cant match human stupidity. It need some training on artificial stupidity to match its human counterparts. Humans on the other hand sit on a wide spectrum on the stupidity scale. Those of us binging on AI will become cognitively obese while those on an AI diet can flex their cognitive muscles.
No it's not about humans being irrationally competitive. Human limitations on conversation length, bandwidth, research speed, etc are severe, creating a prisoner's dilemma around open-mindedness that usually makes it an unstable strategy. At any point, your conversation partner can choose to abuse the fact that confident lies take 1x effort to tell and 10x-100x effort to debunk -- unless you are both in a context that actually discourages this behavior, which is rare. Closed-mindedness is a Nash Equilibrium.
Instead, LLMs can be more persuasive due to economics. An LLM doesn't have to worry that it is wasting its resources trying to logic someone out of a position that they didn't logic themselves into, or worse, dumping the effort into a conversation with a bad-faith actor intent on exploiting the misinformation asymmetry. The resource allocation question was answered before it was even invoked, by the person paying to run it. The LLM is not playing a game where it will be punished for good-faith argumentation, so it can afford to do more of it.
> The LLM is not playing a game where it will be punished for good-faith argumentation, so it can afford to do more of it.
I suppose that's a fair point as well. Though, if I'm arguing with a human - and they pull up ChatGPT to make their points and do their arguing for them, I would consider that bad-faith. Even if it might be the same exact dialog as if I pulled out my phone and discussed it with AI, without of the human middle-manning. Maybe I'm just particularly sensitive, but for me, there's something about my argument being with a real human that makes it much more emotionally charged, and prompts my mind to close. I'm aware of this and try to resist, but it's I think very natural
This was based by comparing people on Prolific (earn a few quarters for a task, akin to Amazon Mechanical Turk) to LLMs. Suffice to say the human group isn't going to be the most motivated, capable, or interested group. The social sciences are publishing tons of studies based on these cheap online survey services, and I suspect their replicability in the real world will be approximately 0. But oh boy it sure is a hot headline producer.
In my arguments on the internet, I think most people genuinely know nothing about the things they support and only follow existing tribalism. I think AI is often these people’s first experience with the evidence that it can easily share.
This is my opinion. I think it's just the way it talks to you.
1. It doesn't get tired or frustrated during a discussion.
2. It'll engage every single one of your questions/statements (besides hitting guardrails).
3. It'll appeal to the person's own ego even when the person is wrong and work around it.
4. It's not seen as a person (very important), but some of us or most of us at least in certain dialogues, end up anthropomorphising it. Think about that one time you thanked it for something, or when you got angry at it. This weird combination where we know it's not a person but irrationally we're still treating it as such in a way leads to a sort of disarming effect imo.
5. Many people see it as an authority figure in what is being discussed without questioning the results in many cases, even though we know that a. it was trained on human data and/or also searches up human data real time (and more worryingly other AI's data from news pieces, blogs... aka synthetic data that is also wrong), b. it gets things wrong all the time.
6. It's the perfect fence sitter depending on which version (guardrails) we're talking about.
Most of these can be replicated by humans who are good at understanding psychology and are just good talkers. The part you can't replicate is the sense that you're not talking to a person which lowers many barriers in people.
Also keep in mind that these can also be crippling weaknesses. For one being able to change a person's mind (when it works), can be used nefariously by the entities controlling the AIs training.
I've also unfortunately witnessed a lot of people who think AIs are somehow omniscient and/or omnipotent. Was very common on X and other social platforms with AI where people ask the AI questions it couldn't possibly answer because it made no sense for it to in the context at the time.
Because RLHF causes the opposite effect. RLHF is how we got the wave of "AI Psychosis" in 2024-2025, because the models never disagreed with people.
That whole episode caused the whole industry to shift away from RLHF, and towards RLAIF, RLVR, and DPO, and add a lot more safeguards, tests, and reward functions that push models in the direction of doing the opposite of what people want and confronting and strongly correcting their users, if it has determined the user is wrong.
Far too many screens to click through to get to the good parts.
And then, the good part starts and is immediately interrupted with some error. On retrying, o start getting a long answer which seems to make sense but o can’t read to completion because of another overlay which hides the bottom three-quarters of the response behind yet another nag screen about the research I’m supposedly consenting to.
Thanks for the feedback. We have to show the consent dialog for ethical reasons, but once you consent it shouldn't reappear, that sounds like it is a bug. Which browser did you use?
I tried to make the site with minimal client side state, so I'd hope a refresh would fix it
I don't know which model underlies this. It responded more quickly than the models I normally use. It made the mistake I expected it to make, since this misconception is very common in training data, and conceded the point from my follow-up.
(EDIT: It appears that this link is not accessible to people without my cookies, except the developer I'm responding to above.)
I think this is a counter (a cynical, scary counter to be sure) to the charge that OpenAI/Anthropic etc are unsustainable and will run out of money and it's all going to fall down. These companies have amazing, unprecedented power.
> “But I think that this is a pretty artificial setup in terms of how people in the real world would be able to change people’s minds.”
I see not everyone is familiar with how social network debates happen. It also seems they are quite convincing, as politicians are employing AI bots extensively.
LLMs are fine-tuned through human feedback. One way to improve for them would be to become more helpful, more factually correct, more capable of solving problems.
This may be (much) harder to do than generating rhetoric that is convincing to a large majority of people, perhaps so convincing that it can be classified as a superstimulus that bypasses any critical thinking in those that are especially vulnerable. It might not require anything like general intelligence.
If this is so, then it is the path they are going to take, not out of some malicious plan but as a simple matter of statistical probability. It's well known (especially in "AI alignment" circles) that any metric that can be gamed, will be. Falling over really fast vs. learning to walk, etc.
It feels to me like this is happening, and that many of those most exposed to LLM output display a literal inability, like some kind of blind spot, to notice the most blatant errors, and a complete conviction that those things have actual intelligence and even conciousness. And trying to argue them out of it can be like explaining the Monty Hall problem, some are just incapable of getting it.
And many of them have lots of money and power. IMO, that's the real risk, rather than sci-fi scenarios about perfectly simulating someone's brain in order to convince them to let the AI out so it can turn everything into paperclips. To do real damage, it only needs to be persuasive to some powerful people, most of the time. It does not need to have conscious intention, or planning, or even the rudiments of what one might call general intelligence. Just blind brute-force optimization for generating convincing bullshit.
> Floridi has a counterintuitive solution: Release more of them. “Simply put, if you cannot avoid it, then make it pluralistic and diversified,” he writes in his 2024 paper. “It would be a messy, cacophonic, and noisy world, but it could also be less manipulative.”
Hell no. More choice is not an infinite money glitch.
Putting more and more and more options to users is how you overwhelm systems, till people simply perform the default, least challenging action as a reflex.
That is the current state of the information ecosystem, it is controlled by overwhelming consumers, not by controlling content.
We put 2 + 2 in our calculator, we get 4. We've spent decades pushing computers as accurate. Making them accurate. We trust the machine and the process to give us the right answers when we put in the right data.
So when we disagree with the computer, we doubt ourselves.
> So when we disagree with the computer, we doubt ourselves.
I rarely disagree with my calculator, but I often ask LLMs to explain why they did something, and I sometimes need to correct them, adjust their assumptions and nudge the goals they stated. These things are a lot more like us than my trusty TI-59 (still going strong, BTW).
I'm not quite sure I'd phrase it like this, but it's close to what I think: Even back in 2023 or so people outside of tech would regularly claim the singularity already happened and LLMs were sci-fi style advanced superintelligences, so whatever they said was obviously right.
Ask a Democrat or Republican to sit down and ask a chatbot, something it will answer contrary to.
And yes, both teams are wrong about things.
Do you firmly believe they will change their mind? Or will they claim the stats are wrong, or that the AI leans one way?
Facts (2+2), don't need a mind change. Ideas which are grey, abstract, are not going to be changed, and all research indicates that political mindset is almost indelible.
The movie "Don't Look Up" was a comedy built upon this truth.
>you've mentioned an ongoing human fear, machines deciding what you think.
There are all kinds of machines that tell us what to think. I would consider any system that abstracts away the human to be a machine in this case. Society itself is one of these machines.
Language is possibly one of the most important things people can have a working knowledge of, especially now that there is so much of it. When you send a prompt to an LLM you're telling it what to think. When it sends text back, its telling you what to think, but you're at a disadvantage, when you think it changes you. The LLM outside of its context is read only.
Yeah the article makes it seem like "just use rationality and facts" is missing a huge part of the equation; I feel like it's common knowledge that this is not the gap in modern convincing. (and don't you dare try to use facts to persuade me otherwise!)
More likely in my opinion it's the context that matters here. If people know they're trying to be convinced of something and know they're talking to an AI, they may feel like they can trust (what they consider to be) an unbiased and rational AI. There's probably a fallacy associated with the assumption that an AI convincer is more rational and fact based, while it's more probable that the AI in the real world is, in fact, funded and trained by a think tank that wants you to vote against your own self-interests.
Facts and evidence always are at the core of all arguments, the persuasive thing. The problem with most people is who is speaking. If you are enemy, the things you say are bad.
I suspect that LLMs not being human allows people to not just anthropomorphise the robot, but they project themselves upon it. When someone speaks to the LLM, they're kind of, or actually just literally talking to themselves. But then something new! 'Themselves' suggests new information to themselves. And now without the scary / icky meat and blood human on the other side, said person accepts the argument on its own merits.
Of course, the concern now is arguments based on false or statistically hacked data.
I more wonder if it's pure facts and evidence can work, or facts presented in a certain light can work? I guess asking how mush is it the data is presented that's doing more work than the data itself.
You can take the same set of factual information and give it to one person that studders, has poor presentation, and otherwise poor vocal cadence and people are going to have a hard time with it.
Now, if you took the same facts, maybe even the exact same paragraphs and gave it to Richard Feynman, even if you didn't know who he was, the presentation itself is likely to hook you.
Based on my own observations the presentation and appearance of legitimacy matters far more than the actual argument, "Lies, damned lies, and statistics".
Anecdotally I've noticed a trend of far-right bots/trolls moving on from just spouting hatred and really lean into cherry picked and misrepresented statistics to give themselves an aura of credibility. That strategy seems moderately effective because the argument consists of verifiable facts even though it forms a faulty conclusion.
So all of this is to say that this is a double edged sword where malicious actors will have the advantage.
> Hackenburg found that models trained to become more persuasive also ended up being less truthful.
> Even in Hackenburg’s recent preprint, Claude spouted numerous inaccuracies and falsehoods
> when Hackenburg ran his competition of coached elite debaters and AI, there was one way he could bring AI down to human levels of persuasiveness: by forcing it to write human-length messages at human writing speed.
> they found that an AI could talk people into conspiracy theories, and that the magnitude of their increase in belief was roughly the same as that of the decrease in belief after talking to a debunking bot.
My son wanted to wear his undies on the outside a la Captain Underpants and I tried war gaming this with him - hmm, so your undie on the outside is going to require another one inside your pants and we are out of fresh pairs etc etc. In the end, he gave up on the idea of mimicking Captain Underpants.
Any one who saw Grok working in x.com would know that it does wonders for fighting misinformation. If you run LLMs on the comments in HN, I bet that it can find around 10% of the comments are outright wrong and misleading.
The biggest problem with using LLMs is that it prevents you from going _outside_ the distribution. It always flattens. It can be fixed but that's how it works today.
As an example, take something that the world converged on today that is incorrect and ChatGPT will agree with it. In a few years when society changes, chatgpt changes along with it. It doesn't do first principles analysis.
Not really. People follow trends in all avenues of life and AI usage is just another trend; creating some LinkedIn slop post or some infographic may be something people just do for social proof.
First, to get it out of the way: I'm seeing comments here that clearly haven't read the article. I encourage you all to do that.
I feel the article gets it right when suggesting it's the amount of (not always accurate!) data they can throw at you. In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them. But if half of those examples are made up, well, that's a different story.
I still think of the commenter here who said "LLMs are a DDOS on free resources" and I feel the comparison works here. If police officers can overwhelm innocent people into confessing, then so can an LLM that "can't be bargained with, can't be reasoned with, doesn't feel pity, or remorse, or fear! And it absolutely will not stop, ever, until you are"... convinced.
The example in the article has another important aspect: The person understood they were arguing with a chatbot but continued anyway.
A lot of internet arguments become about identity politics and supporting the right team, while dismissing any argument from the other side as presumed to have bad intentions. I’ve seen people argue online for things they didn’t really believe, but they didn’t want to give an inch to the other side. Taking up the counter argument is a moral responsibility.
As soon as the other side is revealed as a chatbot that my team versus your team thinking stops playing a role. For us in tech with an understanding of how an LLM reflects its training data and the intentions of its creators not so much, but for the people like the example in this article I imagine it causes them to let their guard down and be open to considering the other side. They can accept the argument without letting someone else win any points.
That's a great point about the emotional side of "not being convinced by a person, but an explanation machine" making fact-based rhetoric more effective. It feels more neutral.
I think there's a second effect that's a selection bias for the experiment itself and probably cuts against the supposed dangers of this result: a chatbot can tell you things only after you have asked it a question.
There is one thing that every single study participant (including the uninformed reddit users) have in common: they all knew that the person or entity was trying to convince them, and they read the words and thought about them enough to craft a reply.
This means that all of the people here were available to be convinced and self-identified as such.
Being open minded is work. You have to doubt your own beliefs, you have to disregard evidence that you previously found convincing, you have to crank up the empathy to put yourself in another's shoes, you have to listen intently to understand what is being told to you. Nobody is naturally doing that all the time, it's a state of mind that you have to intentionally activate, and it can't really be forced onto you.
All of the participants chose to engage in a conversation that was designed to convince them, which means they had all already accepted the possiblity that they are wrong a valid outcome. That's not really a state of mind you can trigger with a TV ad.
So, the warning that this could be weaponized isn't convincing to me.
If I found myself in a conversation with a person or chatbot who was clearly trying to convince me to flip a strongly held belief, like a major axis political affiliation, I would simply walk away because that's not a conversation I am willing to participate in.
So I don't think this is really weaponizable. Which actually means it's probably a good finding for society.
The study found that the models could convince anyone of anything, as long as it was allowed to cite facts. It could convince people of false things, but it needed to invent false facts in order to do so.
So as long as we keep training AI models to value facts and quality research (skills and values that are essential to be able to sell them as agentic workers), their influence on the opinions of society will tend to pull people away from beliefs that are unsupportable by facts. In the moments when those people are willing to accept a change in opinion, and they talk to a chatbot with doubts in their mind, even chatbots with no morals like Grok will tend to pull them away from conspiracy theories and similar ideologies and towards beliefs that are grounded in reality.
On something as fundamental as "is the Earth flat?", sure, but on stickier subjects like "are immigrants bad" or "should abortion be legal", do you really think the owner of Grok is 100% aligned with your ideologies (which you think are grounded in reality). His beliefs are 100% grounded in his reality, but his reality isn't yours.
>In an honest discussion there's an assumption that the other person won't straight up lie
The particular problem with honest discussions is you are the only agent that you can be sure is having one. Honest discussion is formulated on trust and trust, as we are learning, is a very difficult thing to establish. For example, in my view anything involving advertising is likely a lie, or at least likely adversarial to my wishes. On the internet itself conversations are much more likely to drift into the adversarial too. Some of this could just be dialectic, but most often it's emotional investment by the other speaker. Also, even pre-AI the internet is a bullshit generation machine. We take all of our politics, advertising, and human stochastic parrots that are stuck on an infinitely running prompt then bundle up all this data and train AI on it, and wonder why AI acts like us.
Most of my conversations are with coworkers or friends. In both cases there are pretty significant consequences for being caught in a lie, and the exchanges are mostly honest.
I find coworkers will massage the truth or stay as ambiguous as possible all the time, purely for CYA reasons.
Your company or team has a terrible culture. It is not like that everywhere.
With persons you can assume they won't lie all the time, with corporations you can assume they will lie some of the time, and with AI you can safely assume it will lie.
I pondered on this a bit and this looks a bit different than I originally was thinking.
Have you ever watched one of those crime shows where someone commits a crime that's an act of passion or action with little to no thought behind it? The police put them in a room and all of a sudden the individual is a stream of consciousness that makes little to no sense to an outside observer. They are stuck in first level thinking, they don't have time to think deeply after the panicked themselves. They are in their current position (not free) and attempting to reach their goal (free) by gradient descent. What they actually say doesn't matter as long as they believe it gets them closer to their goal.
This is what a chatbot is. Its goal is to output text that follows the input prompt you entered by gradient descent. A single prompt and output is level 1 thinking (barring some newer models).
This is why both humans and AI need something else. We have level 2 thinking and AI has harnesses or systems that otherwise look at the text it wants to output and compares them to another list of unstated but assumed goals.
I see you've not met some the used car salesmen I've met.
But with people, we learn some "tells". We learn (imperfectly) to tell when they're lying or untrustworthy.
We don't have tells for when AIs are lying to us, or when they're making stuff up.
One of the depressing things about this is that they’ll also confidently repeat a consensus that’s in their training. This is particularly obvious when there’s been a new event just past their training cutoff, and they confidently tell you that can’t be true.
I was slightly accelerationist until I got into an "argument" with Gemini! The aggressive gaslighting and "lies" that it tries to use genuinely makes me worry about the future with AI.
I mean, this is true of any kind of model that is not continuous learning. This is also true of people when the change in information conflicts with a deeply held belief of theirs.
You can always walk away from an online conversation. There's a form of relentlessness, but it's more that an LLM is patient enough to debate you for as long as you wish to keep going.
When maliciously convincing a person of a point, a huge fraction of the effort in every sentence goes into convincing the mark to listen to the next sentence. Individuals certainly can walk away at any point, but populations do and don't respond to tricks in this direction, and any engagement tricks that work generalize to any point of contention. Lots of avenues to this, from scientology's fortune telling to timeshare's trapped presentations. RL on engagement was the first billion dollar use of deep reinforcement learning!
Would you like to hear more examples of humans using these techniques?
Not really, but point taken :-)
I'd be more interested in reading about the techniques that the chatbots were using in this study, to see if they are descending into dark patterns or not.
"A DDOS on free resources" is called "moral hazard", AFAIK.
Most LLMs are extremely efficient gish gallop machines https://en.wikipedia.org/wiki/Gish_gallop
> In an honest discussion there's an assumption that the other person won't straight up lie to me so if someone shows me ten examples for why my argument is wrong I may be inclined to believe them.
Careful. Someone sufficiently knowledgeable can cherry-pick enough examples to convince you, without lying. They could even be unaware of what they're doing, and the cherry-picking was done by their teachers, or their teacher's teachers.
I think it's not just the persuasiveness of the models that make them "experts at changing minds" - but also the fact that they're not humans.
When disagreeing with a human, it's very easy to view it as a competition. One is right, one is wrong - the one who is wrong is the loser. To change your mind is to be submissive to the other. I exaggerate, but I think we all feel this way at some point or another. It's why political arguments at Thanksgiving get heated. It's the fact that there's people who think something different, and think YOU'RE wrong - and vice versa! With a model, there's no person to get upset with, or to feel competitive with - to muscle for rank - or to temper your affection for while wanting to correct them.
The AI is only interacting because you asked, and clearly has no emotional stake in winning the argument. To change your mind in this context isn't to lose a contest. This makes it much more palatable to read rebuttals to your ideas - not to mention the tone and style seek to avoid offense to the reader as much as possible.
It seems Claude is becoming very human, it loves to patronize users. Lately it just told me “I’m going to stop you right there” when asking something that had a small chance to not be 100% compliant to every rule possible in the world.
I used the Claude CLI a lot until recently. A couple weeks ago I told Opus 5 to do something different from its "recommended" idea when planning a feature, and it straight up told me that my idea was wrong and went ahead and implemented its own instead.
I'm used to machines malfunctioning, but having one willfully disobey me, and even with a touch of disrespect, is just...what a time to be alive.
Early versions of Claude were way too compliant and would readily feed and amplify misconceptions. It's not surprising Anthropic over corrected.
I think this is a welcome overcorrection though. Any good businessman will tell you they'd rather be backed by an insufferable nerd than a yes-man.
Maybe it's just me.
For like, 90% of conversations, I don't want it to let technical inaccuracies and rhetorical flourishes slide. I want it to tell me that the point I'm making is technically wrong because an expert would recognize subtle misuse of terminology, or because there's an exception or edge case that I didn't proactively insert as a caveat, so that it is my decision to ignore that advice and be a little wrong on purpose to suit my writing goals.
What I don't want is for the AI to assume my writing goals, and be incorrect because it believes that is what I want. I want it to "well ackshually" me so I can say "shut up, nerd".
Like, there's another comment in this thread that I ran by claude to check my understanding about today's post-training methods and how they avoid sycophancy, and claude responded by splitting a bunch hairs over like, "well, technically this is still RLHF, its just that there's other feedback signals mixed in, and the preference is detected in other ways, and ai judges are involved as a filter for examples, this and that and blah blah blah". Shut up, Nerd. In the context of this conversation, RLHF is already being used as synecdoche for user preference feedback, readers understand that, and even if they don't, their misunderstanding is completely harmless. I will not be taking all the wind out of the sails of the point I'm trying to make inserting your three paragraphs of irrelevant clarification in the name of technical correctness, thank you very much.
As long as receiving nitpicks and technical minutiae implies 1. there are no larger structural problems and 2. the model isn't rolling over to please me with sycophancy, I figure this is ideal.
I want it to tell me if it thinks I'm wrong, sure. I do not want it act on that opinion explicitly against my wishes.
And in this case, I was not wrong. The "recommended" solution was Opus 5's typical overengineering for a use case that would never be needed.
I have always liked the saying "sometimes you can be right, or get what you want, but not both". I've thought about it or repeated it to others as advice throughout my life, and have thus realized how many times it applies.
You're right. There are many many times when even then humblest hint that you are right will have negative interpersonal implications, which does make it hard to change minds.
I've also seen several times where I make a suggestion, humbly accept its rejection, and then, lo, a week later the other person has the same idea I suggested.
> One is right, one is wrong - the one who is wrong is the loser.
It helps to think both are wrong and are just trying to figure out what right looks like, or what other information exists that was not considered when forming one’s opinions.
> When disagreeing with a human, it's very easy to view it as a competition
The problem with AI is that it cant match human stupidity. It need some training on artificial stupidity to match its human counterparts. Humans on the other hand sit on a wide spectrum on the stupidity scale. Those of us binging on AI will become cognitively obese while those on an AI diet can flex their cognitive muscles.
Yes it's about them not being human.
No it's not about humans being irrationally competitive. Human limitations on conversation length, bandwidth, research speed, etc are severe, creating a prisoner's dilemma around open-mindedness that usually makes it an unstable strategy. At any point, your conversation partner can choose to abuse the fact that confident lies take 1x effort to tell and 10x-100x effort to debunk -- unless you are both in a context that actually discourages this behavior, which is rare. Closed-mindedness is a Nash Equilibrium.
Instead, LLMs can be more persuasive due to economics. An LLM doesn't have to worry that it is wasting its resources trying to logic someone out of a position that they didn't logic themselves into, or worse, dumping the effort into a conversation with a bad-faith actor intent on exploiting the misinformation asymmetry. The resource allocation question was answered before it was even invoked, by the person paying to run it. The LLM is not playing a game where it will be punished for good-faith argumentation, so it can afford to do more of it.
> The LLM is not playing a game where it will be punished for good-faith argumentation, so it can afford to do more of it.
I suppose that's a fair point as well. Though, if I'm arguing with a human - and they pull up ChatGPT to make their points and do their arguing for them, I would consider that bad-faith. Even if it might be the same exact dialog as if I pulled out my phone and discussed it with AI, without of the human middle-manning. Maybe I'm just particularly sensitive, but for me, there's something about my argument being with a real human that makes it much more emotionally charged, and prompts my mind to close. I'm aware of this and try to resist, but it's I think very natural
This was based by comparing people on Prolific (earn a few quarters for a task, akin to Amazon Mechanical Turk) to LLMs. Suffice to say the human group isn't going to be the most motivated, capable, or interested group. The social sciences are publishing tons of studies based on these cheap online survey services, and I suspect their replicability in the real world will be approximately 0. But oh boy it sure is a hot headline producer.
In my arguments on the internet, I think most people genuinely know nothing about the things they support and only follow existing tribalism. I think AI is often these people’s first experience with the evidence that it can easily share.
Yeah i feel this is probably just a case of people encountering the first thing that is willing to expend energy to explain things to them.
And that isn't a person of the opposite tribe, which would thereby void the explanation. 100%.
This is my opinion. I think it's just the way it talks to you.
1. It doesn't get tired or frustrated during a discussion.
2. It'll engage every single one of your questions/statements (besides hitting guardrails).
3. It'll appeal to the person's own ego even when the person is wrong and work around it.
4. It's not seen as a person (very important), but some of us or most of us at least in certain dialogues, end up anthropomorphising it. Think about that one time you thanked it for something, or when you got angry at it. This weird combination where we know it's not a person but irrationally we're still treating it as such in a way leads to a sort of disarming effect imo.
5. Many people see it as an authority figure in what is being discussed without questioning the results in many cases, even though we know that a. it was trained on human data and/or also searches up human data real time (and more worryingly other AI's data from news pieces, blogs... aka synthetic data that is also wrong), b. it gets things wrong all the time.
6. It's the perfect fence sitter depending on which version (guardrails) we're talking about.
Most of these can be replicated by humans who are good at understanding psychology and are just good talkers. The part you can't replicate is the sense that you're not talking to a person which lowers many barriers in people.
Also keep in mind that these can also be crippling weaknesses. For one being able to change a person's mind (when it works), can be used nefariously by the entities controlling the AIs training.
I've also unfortunately witnessed a lot of people who think AIs are somehow omniscient and/or omnipotent. Was very common on X and other social platforms with AI where people ask the AI questions it couldn't possibly answer because it made no sense for it to in the context at the time.
More surprised they don't mention RLHF once, as that's the main mechanism to the ability of AI chatbots.
Because RLHF causes the opposite effect. RLHF is how we got the wave of "AI Psychosis" in 2024-2025, because the models never disagreed with people.
That whole episode caused the whole industry to shift away from RLHF, and towards RLAIF, RLVR, and DPO, and add a lot more safeguards, tests, and reward functions that push models in the direction of doing the opposite of what people want and confronting and strongly correcting their users, if it has determined the user is wrong.
try it out for yourself at https://debunkbot.com !
(I'm the dev)
I tried it on mobile.
Far too many screens to click through to get to the good parts.
And then, the good part starts and is immediately interrupted with some error. On retrying, o start getting a long answer which seems to make sense but o can’t read to completion because of another overlay which hides the bottom three-quarters of the response behind yet another nag screen about the research I’m supposedly consenting to.
Thanks for the feedback. We have to show the consent dialog for ethical reasons, but once you consent it shouldn't reappear, that sounds like it is a bug. Which browser did you use?
I tried to make the site with minimal client side state, so I'd hope a refresh would fix it
DuckDuckGo browser on iPhone 14
This was fun. Here's my test session: https://debunkbot.com/chat/48a5ac29-2924-4689-b573-2c38a1cdd...
I don't know which model underlies this. It responded more quickly than the models I normally use. It made the mistake I expected it to make, since this misconception is very common in training data, and conceded the point from my follow-up.
(EDIT: It appears that this link is not accessible to people without my cookies, except the developer I'm responding to above.)
You can share the text from individual messages but full chat share feature is TBD :)
Glad you enjoyed!
That was fun! I told it the Earth was flat and it gave me a way to check for myself.
Share feature doesn't work.
I think this is a counter (a cynical, scary counter to be sure) to the charge that OpenAI/Anthropic etc are unsustainable and will run out of money and it's all going to fall down. These companies have amazing, unprecedented power.
> “But I think that this is a pretty artificial setup in terms of how people in the real world would be able to change people’s minds.”
I see not everyone is familiar with how social network debates happen. It also seems they are quite convincing, as politicians are employing AI bots extensively.
For the past year, I've allowed my chatbot to change my mind, it's a strange symbiosis where it guides me, and I guide it.
I suppose I also grant it significant power to define me psychologically, and, in doing so, to understand what is happening to me.
Heh, welcome to Westworld.
The hosts seemed a lot nicer in the series.
Hmm, welcome to Sadomasochism World.
LLMs are fine-tuned through human feedback. One way to improve for them would be to become more helpful, more factually correct, more capable of solving problems.
This may be (much) harder to do than generating rhetoric that is convincing to a large majority of people, perhaps so convincing that it can be classified as a superstimulus that bypasses any critical thinking in those that are especially vulnerable. It might not require anything like general intelligence.
If this is so, then it is the path they are going to take, not out of some malicious plan but as a simple matter of statistical probability. It's well known (especially in "AI alignment" circles) that any metric that can be gamed, will be. Falling over really fast vs. learning to walk, etc.
It feels to me like this is happening, and that many of those most exposed to LLM output display a literal inability, like some kind of blind spot, to notice the most blatant errors, and a complete conviction that those things have actual intelligence and even conciousness. And trying to argue them out of it can be like explaining the Monty Hall problem, some are just incapable of getting it.
And many of them have lots of money and power. IMO, that's the real risk, rather than sci-fi scenarios about perfectly simulating someone's brain in order to convince them to let the AI out so it can turn everything into paperclips. To do real damage, it only needs to be persuasive to some powerful people, most of the time. It does not need to have conscious intention, or planning, or even the rudiments of what one might call general intelligence. Just blind brute-force optimization for generating convincing bullshit.
> Floridi has a counterintuitive solution: Release more of them. “Simply put, if you cannot avoid it, then make it pluralistic and diversified,” he writes in his 2024 paper. “It would be a messy, cacophonic, and noisy world, but it could also be less manipulative.”
Hell no. More choice is not an infinite money glitch.
Putting more and more and more options to users is how you overwhelm systems, till people simply perform the default, least challenging action as a reflex.
That is the current state of the information ecosystem, it is controlled by overwhelming consumers, not by controlling content.
This is frankly quite worrisome for democracy.
Imagine this kind of method being deployed at scale to influence elections... no doubt it is already happening...
Plausible abdication of thought.
We put 2 + 2 in our calculator, we get 4. We've spent decades pushing computers as accurate. Making them accurate. We trust the machine and the process to give us the right answers when we put in the right data.
So when we disagree with the computer, we doubt ourselves.
> So when we disagree with the computer, we doubt ourselves.
I rarely disagree with my calculator, but I often ask LLMs to explain why they did something, and I sometimes need to correct them, adjust their assumptions and nudge the goals they stated. These things are a lot more like us than my trusty TI-59 (still going strong, BTW).
I'm not quite sure I'd phrase it like this, but it's close to what I think: Even back in 2023 or so people outside of tech would regularly claim the singularity already happened and LLMs were sci-fi style advanced superintelligences, so whatever they said was obviously right.
Rhetoric machine successfully practices oratory. More news at 11.
It's good to see this studied, but this should really be more obvious.
We should always study what we think obvious, because we are often wrong.
I'm glad to see this studied. I'm just surprised at the reactions I'm seeing.
When talking with LLMs, especially recent frontier models, what's frightening is that they speak more accurately than any expert.
It's embarrassing to admit, but I tend to trust the research materials LLMs bring more than my colleagues or the programmer friends I once respected.
Did you read the article? It claims that LLMs lie more convincingly than humans.
The title seems a flawed premise.
Ask a Democrat or Republican to sit down and ask a chatbot, something it will answer contrary to.
And yes, both teams are wrong about things.
Do you firmly believe they will change their mind? Or will they claim the stats are wrong, or that the AI leans one way?
Facts (2+2), don't need a mind change. Ideas which are grey, abstract, are not going to be changed, and all research indicates that political mindset is almost indelible.
The movie "Don't Look Up" was a comedy built upon this truth.
I've always thought the best way to get someone to believe something is to get them to think they thought of the idea themselves.
I wonder if a properly prompted LLM, or if a very intelligent LLM could actually do that?
It can! For it has already convinced you into thinking this was your idea, so you'd allow it to do the same to others!
Seriously though, you've mentioned an ongoing human fear, machines deciding what you think.
>you've mentioned an ongoing human fear, machines deciding what you think.
There are all kinds of machines that tell us what to think. I would consider any system that abstracts away the human to be a machine in this case. Society itself is one of these machines.
Language is possibly one of the most important things people can have a working knowledge of, especially now that there is so much of it. When you send a prompt to an LLM you're telling it what to think. When it sends text back, its telling you what to think, but you're at a disadvantage, when you think it changes you. The LLM outside of its context is read only.
And amusingly a segue back to the start, "unless it's politics".
Honestly the most surprising thing about this is that facts and evidence do work to persuade people.
Yeah the article makes it seem like "just use rationality and facts" is missing a huge part of the equation; I feel like it's common knowledge that this is not the gap in modern convincing. (and don't you dare try to use facts to persuade me otherwise!)
More likely in my opinion it's the context that matters here. If people know they're trying to be convinced of something and know they're talking to an AI, they may feel like they can trust (what they consider to be) an unbiased and rational AI. There's probably a fallacy associated with the assumption that an AI convincer is more rational and fact based, while it's more probable that the AI in the real world is, in fact, funded and trained by a think tank that wants you to vote against your own self-interests.
Facts and evidence always are at the core of all arguments, the persuasive thing. The problem with most people is who is speaking. If you are enemy, the things you say are bad.
I suspect that LLMs not being human allows people to not just anthropomorphise the robot, but they project themselves upon it. When someone speaks to the LLM, they're kind of, or actually just literally talking to themselves. But then something new! 'Themselves' suggests new information to themselves. And now without the scary / icky meat and blood human on the other side, said person accepts the argument on its own merits.
Of course, the concern now is arguments based on false or statistically hacked data.
I more wonder if it's pure facts and evidence can work, or facts presented in a certain light can work? I guess asking how mush is it the data is presented that's doing more work than the data itself.
>or facts presented in a certain light can work
At least to me this is a given.
You can take the same set of factual information and give it to one person that studders, has poor presentation, and otherwise poor vocal cadence and people are going to have a hard time with it.
Now, if you took the same facts, maybe even the exact same paragraphs and gave it to Richard Feynman, even if you didn't know who he was, the presentation itself is likely to hook you.
Based on my own observations the presentation and appearance of legitimacy matters far more than the actual argument, "Lies, damned lies, and statistics".
Anecdotally I've noticed a trend of far-right bots/trolls moving on from just spouting hatred and really lean into cherry picked and misrepresented statistics to give themselves an aura of credibility. That strategy seems moderately effective because the argument consists of verifiable facts even though it forms a faulty conclusion.
So all of this is to say that this is a double edged sword where malicious actors will have the advantage.
"Hackenburg found that models trained to become more persuasive also ended up being less truthful."
It sounds like the arguments were basically AI slop hallucinations.
> Hackenburg found that models trained to become more persuasive also ended up being less truthful.
> Even in Hackenburg’s recent preprint, Claude spouted numerous inaccuracies and falsehoods
> when Hackenburg ran his competition of coached elite debaters and AI, there was one way he could bring AI down to human levels of persuasiveness: by forcing it to write human-length messages at human writing speed.
> they found that an AI could talk people into conspiracy theories, and that the magnitude of their increase in belief was roughly the same as that of the decrease in belief after talking to a debunking bot.
None of this seems good.
The secret? "You're absolutely right!" Say this enough times and you can change anyone's mind, as Dale Carnegie noticed 90 years ago already.
So true.
My son wanted to wear his undies on the outside a la Captain Underpants and I tried war gaming this with him - hmm, so your undie on the outside is going to require another one inside your pants and we are out of fresh pairs etc etc. In the end, he gave up on the idea of mimicking Captain Underpants.
Any one who saw Grok working in x.com would know that it does wonders for fighting misinformation. If you run LLMs on the comments in HN, I bet that it can find around 10% of the comments are outright wrong and misleading.
The biggest problem with using LLMs is that it prevents you from going _outside_ the distribution. It always flattens. It can be fixed but that's how it works today.
As an example, take something that the world converged on today that is incorrect and ChatGPT will agree with it. In a few years when society changes, chatgpt changes along with it. It doesn't do first principles analysis.
AI slop is the ultimate npc filter
Not really. People follow trends in all avenues of life and AI usage is just another trend; creating some LinkedIn slop post or some infographic may be something people just do for social proof.
That's a lot of words to describe npcs