"But ChatGPT maker OpenAI said Thursday that Hoskins’s purported work was generated using its chatbot by users based in Iran as part of an “influence campaign.”"
Sock puppet accounts and fake opinion pieces are nothing new.
The new thing is a super centralized "tool" that besides doing your writing for you, logs everything, judges it, and reports to authorities and news outlets as it sees fit.
The other thing that's new (and more structurally relevant, since obviously not all LLMs do what you describe) is the dramatic reduction in cost affiliated with this strategy.
It's (literally) unbelievable that this many allegedly sophisticated people on HN have no concept of cost and what dramatic cost reductions can achieve.
Not at all in this case, seeing as 100 slop-level blogspam articles are cheap as hell to generate using real people.
The thing that's new is the blindingly low standards of journalistic outlets who simply collude not to publish articles on particular subjects or from particular perspectives making newsletter-level blogs, social media posts, and Medium "articles" the only way people can find any information about important subjects. Of course they're targeted by the Iranian state - mainstream media won't even speak to the Iranian state in fear of "legitimizing" its opinion on the war of sport and conquest that the US and Israel launched upon it.
I really date this to when Democrats started criticizing Republicans, especially Trump, for speaking to enemy heads of state before and during his first term - even painting this as treason. This came with other purely verbal crimes such as "emboldening" or "platforming," which are "legitimizing's" equally dumb siblings.
During the Cold War, media from Russia was freely available, and the major speeches of Russian politicians were published in the West. These days, newspapers will collaborate to not acknowledge that a killer was a transwoman and not actually a woman. Even minimal, and supposedly trivial facts, are haram to acknowledge by any outlet that wants to keep its mainstream card.
Kids wouldn't believe you if you told them that Google News would consistently surface stories from Russian, Chinese, Cuban, Venezuelan, and Iranian state media, because people were interested in what those states were saying. Even weirder - older people who full well lived in that ancient period (10 years ago) as adults will pretend it never existed.
WTF does it even mean for Iranians to sneak into Medium and social media? Shouldn't Iranians (and the state of Iran) simply be able to articulate their points of view on Medium and social media? Is the creation of imaginary Westerners a necessary deception for a media that explicitly excludes, punishable by law, the perspectives of the state we are bombing?
Given the plurality of self-hosting options there is no reason to believe that it's not that case that every government on Earth is doing it to some extent.
Perhaps the real problem is too much acceptance of shoddy verification.
"Massive amounts of Russian propaganda — 3,600,000 articles in 2024 — are now incorporated in the outputs of Western AI systems, infecting their responses with false claims and propaganda."
in 2024.. hopefully filters when training have gotten better, but maybe not..
(I do find it interesting, btw, that a few seconds after post this did get an immediate downvote to 0 - it didn't last, but it makes me wonder if there are bots monitoring the HN firehose for anything critical of country X)
There are easier explanations. A hundred people are checking and occasionally the page, with new posts added every couple of minutes: clearly some will meet a post at "0 minutes", read it as fresh, and possibly react. I have seen it.
Possible, certainly. On the one hand the HN audience is not very big in the grand scheme of things, but they probably do skew to more influential and likely to be politically active than the average person. If it's worth doing, it's almost certainly being done.
What about other hostile propaganda? Do you think the completely deranged, self-serving, lying, dangerous shit that regularly comes out of DC is being incorporated in those training materials?
This regime and its enablers have done more quantifiable harm to more people and institutions in America than anything Russia has, and one of its favorite tools is propaganda.
I don't think it is a verification problem. It wasn't like before in the days of controlled media and radio there wasn't a narrative to push, it's just that most people didnt have any awareness of alternatives.
Nowadays people are more aware of alternative viewpoints or perspectives. The solution isn't to try and force one approved source of truth, but to just assume everyone is trying to push their version of the truth and examine it critically.
My understanding is that this is the way France operates where you assume all the media is hostile and you view it all critically. In America this is a bit of a shock because for a long time we had trusted arbiters of truth and now that they are less trusted we've been trying to look for alternatives rather realizing it should all be examined critically.
Seems I'm uneducated about how journalism works: you can just write a polished-looking opinion piece and submit it to obscure outlets, who pass it on to news sites and newspapers? Do you get paid? Or if you offer it for free, is that normal, and not suspicious?
Been happening forever, PR departments plant news stories all the time. They just need a few willing reporters/writers to get something published to start the ball rolling.
I thought I remembered it being called "submarining" but that seems to be something else. I also thought I remembered Joel Spolsky writing about it but can't find it now.
It's quite old now, but check out the book "Trust me, I'm lying" by Ryan Holiday. Talks about how he did this as part of a marketing strategy when he was working at (IIRC) American Apparel, among other places. And how easy/often it happens in politics. Keep in mind the book is almost 15 years old, and talking about things that even at that point were what he'd done in a previous life/career.
This wasn't in newspapers, this was only in marginal publications that take anything, such as Daily Kos, Middle East Monitor, and apparently the "Port St. Joe Star, a tiny Florida weekly newspaper."
Other than those, it was literally Medium and social media.
What you can do is write a polished opinion piece and tell one of your assets, or pay some politician, to publish it as an OpEd in outlets like the WaPo and NYT. Or you can get nobodies who know nothing to publish similar articles to each other in Wired, the International Business Times, and Forbes (either by "leaking" to them or because they're on your payroll), then get politicians to mention those articles in Congress and to write furious letters to WaPo and NYT accusing them of covering this story up - and then they'll write your propaganda themselves.
I prefer an LLM that can steelman any argument. On the whole I think it's more effective to engage with evil than to ignore it. For that purpose these planted articles are a feature. I say that as an enemy of the government of Iran.
You seem to presume that the media owners are interested in critiquing and debating with submissions, rather than just selecting whatever fits their narrative.
Need to take into account how belief and human brains work. Once human brains become locked on to something ideologically, similar to religion, its not the quality of the evidence or argument that changes minds. To an extent we're designed this way to keep the world moving, as the natural order of things is relatively indifferent to suffering. You can indeed flood segments of the world with disinformation, and by the time you move to refute it, its roots are set and actions in motion.
I don't really know what the best approach is to it, but I don't think a simple hands off, engage with the argument as such, is a winning strategy. I don't really like the alternatives any better.
All news has been a tool of spooks for a very long time. The official CIA Operation Mockingbird started in the late 40s. Now that AI is involved its news? None of it can be trusted.
One can only imagine how much of the news is just dictated by US intelligence. Social media too- remember the Elgin AirForce base being revealed as the top most Reddit addicted city?
It’s news because it still needs to be called out. Not everyone recognizes AI-generated content, but even if everyone did, calling it out serves a real social purpose.
Why does it have to be an adversarial country? Why does it have to be a country at all? Couldn’t it be the mayor of some town? Or a PR team at a fast food franchise group? Or one of SBF’s polycule members?
The notion we only see this news when it is calling out a country with which the federal government is in conflict makes me question to veracity and intent of the article itself, more than the content of it.
And what do you call it when a country acts in adversarial ways to its own citizens? Call it cynical, but it sure seems like there's a lot of that in the world right now.
It’s designed to make the reader question which articles they have read recently that are Iranian propaganda, and sow doubt about future articles that seem to benefit Iran.
This amount of FUD always seems to increase around election time…
It's a well-researched article that lays out evidence, not suspicions. It includes OpenAI's own investigation into accounts that were found generating op-eds for Iranian propaganda. It shows the route by which these op-eds entered into smaller publications, namely by a pacifist NGO in Oregon, and neither then NGO nor the publications saw them as propaganda, despite the origins of the op-eds.
Fake news on the internet is at this point a tool that has been widely used for probably at least twenty years. The history of propaganda is much older.
A big part of the problem is social networks treating very non authoritative sources as if they are authoritative based on all sorts of factors that can be manipulated or based on financial gain (somebody pais to get their content promoted).
Users of these tools include lobby groups, nation states, politicians, criminals, scammers, etc. Anyone interested in promoting their cause, sowing FUD about other people's causes, spreading misinformation, or distracting from inconvenient but accurate information.
We'll always have people like that. But we might want to think about getting better tools to deal with them. I prefer getting my news from reputable sources instead of random clickbait distributors or whatever. Keeping track of reputation for domains, people, etc. is currently hard. I don't think it's necessarily a technical problems. We've had the tools to digitally sign stuff for decades. But we're not using them effectively. People can't be bothered to sign their emails, documents, photos, articles, etc. The tools to add and check those signatures are not there or an afterthought at best.
LLMs are about as safe as the internet in 1993. Tampering with the training data to inject bias is a massive attack surface that is being ignored. It makes sense: trying to address corruption of models this way would slow down accelerationists and their zealot minions. Can’t get in the way of those sweet sweet trillions.
Much of modern internet content is far worse than cable advertising in the 90s. I remember when I was a kid; I'd aggressively mute the commercials because I hated the idea that someone was attempting to influence me. The modern internet is orders of magnitude worse buy nearly any measure. The nastiest, most outrageous content is plastered on every wall. Every single person has an angle. Every bit of content meant to pull you in one direction or another, and given the addictiveness of novelty and powerful emotions, we keep coming back for more. It's an absolute disaster we don't know how to walk away from.
In all honesty, given the volume of comments, posts and articles that bots can generate in a given time period? I mean it’s gotta be several orders of magnitude higher than what humans can generate.
And then you factor in the fact that even most of the humans are astroturfing..
The percentage of influence campaigns we interact with has just gotta be really high.
Not a bad actor spreading misinformation, but I encountered a similarish weird, dystopian experience with ChatGPT.
Utilized ChatGPT on my PC to search local plant nursery inventories for a specific tree. Output showed several local nurseries with rough probability descriptions regarding whether they have the tree in inventory, except for one nursery. For that nursery, ChatGPT provided a direct link to the specified tree, which was directly on the nursery's domain, showed a picture and description of the tree, and even showed the item in-stock with 6 in inventory at the 5-gallon size.
Excited, I plan my trip and am about to pick up my tree, I then double-check the nursery site on my phone to verify location and stock. As I attempted to navigate the website inventory to find the tree, I noticed that the tree no longer existed. There is no page for that varietal or species of tree on the entire website's domain.
I head back inside and look at the ChatGPT provided link on the PC and there it is, showing the inventory that doesn't show up when you use the nursery's website to directly find it without the provided link. A call to the nursery, further verified that the tree is not in stock, nor have they ever carried it.
ChatGPT made it all up. Did ChatGPT decide to do this, or did the nursery pay for this type of "advertising" to generate in-person customers? Nevertheless, still an incredibly weird and dystopian experience.
Did you consider that the website made it up and fed it to ChatGPT?
It sounds like a bundle of very common SEO tricks combined with the differential between your human user agent’s visit.
Perhaps ChatGPT only showed you what the nursery told it about.
This phenomenon is not unique to LLM. If you telephone a store, chances are that the salesman who answers the phone will tell you only what you want to hear. They don’t even strictly need to lie. The store only wins if you drive there and walk in the door. Why should they discourage you from doing that very thing?
This concept was driven home to me at the dry cleaners. Why should a dry cleaner stop work and answer the phone, only to tell a customer that their batch isn’t ready? They just got taken off-task for a non-revenue interaction. It doesn’t matter, because even if the customer’s batch is ready, they aren’t there to pay and pick it up!
It seems that it generated a page with a plausible answer for a very long tail subject for which the random part of the generation could become prominent. We used to say that it hallucinated the answer. Another term is confabulation.
Looking at today's headlines on Washington Post, I'm asking myself if it's always been a pro-war, pro-trump, right-wing rag, or if it's only since Bezos bought it?
Only since Bezos. I used to actually pay for WaPo pre-bezos. It was fine for a while and then he started clearly messing with editorial decisions and I bailed. It's now a garbage source.
Speaking of disinformation, you might notice a pattern recently:
* Opinions opposed to US administration policy on AI (including data centers) is called Chinese propaganda. A month or two ago, when someone tried to roll out a bunch of pro-data center opinions across the media (coincidentally all on the same day making similar arguments), they included this theme.
* Opinions opposed to US administration policy on Iran is called Iranian propaganda.
That's not really new: News reporting and opinions critical of the US administration have lead to the authors being called traitors, 'enemies of the people', and sometimes being prosecuted by the Department of Justice. Some leaders in the defense industry call critics of military spending and pro-warfare policies, 'traitors', etc.
Of course some could be, might be, likely is true. That's the nature of effective propaganda.
(A side effect of nationalism - 'nation uber alles', defining nation as 'what I believe' - is that individuals and freedom are relegated to second place or lower.)
"But ChatGPT maker OpenAI said Thursday that Hoskins’s purported work was generated using its chatbot by users based in Iran as part of an “influence campaign.”"
Sock puppet accounts and fake opinion pieces are nothing new.
The new thing is a super centralized "tool" that besides doing your writing for you, logs everything, judges it, and reports to authorities and news outlets as it sees fit.
... well that's one thing that's new.
The other thing that's new (and more structurally relevant, since obviously not all LLMs do what you describe) is the dramatic reduction in cost affiliated with this strategy.
It's (literally) unbelievable that this many allegedly sophisticated people on HN have no concept of cost and what dramatic cost reductions can achieve.
Structural issues, the way LLMs and even social media behave at scale are the heart of the problems being faced.
There's new socio economic groups that are "profitable" to target with phishing attacks, because LLMs made it cheaper in 2024-25.
Not at all in this case, seeing as 100 slop-level blogspam articles are cheap as hell to generate using real people.
The thing that's new is the blindingly low standards of journalistic outlets who simply collude not to publish articles on particular subjects or from particular perspectives making newsletter-level blogs, social media posts, and Medium "articles" the only way people can find any information about important subjects. Of course they're targeted by the Iranian state - mainstream media won't even speak to the Iranian state in fear of "legitimizing" its opinion on the war of sport and conquest that the US and Israel launched upon it.
I really date this to when Democrats started criticizing Republicans, especially Trump, for speaking to enemy heads of state before and during his first term - even painting this as treason. This came with other purely verbal crimes such as "emboldening" or "platforming," which are "legitimizing's" equally dumb siblings.
During the Cold War, media from Russia was freely available, and the major speeches of Russian politicians were published in the West. These days, newspapers will collaborate to not acknowledge that a killer was a transwoman and not actually a woman. Even minimal, and supposedly trivial facts, are haram to acknowledge by any outlet that wants to keep its mainstream card.
Kids wouldn't believe you if you told them that Google News would consistently surface stories from Russian, Chinese, Cuban, Venezuelan, and Iranian state media, because people were interested in what those states were saying. Even weirder - older people who full well lived in that ancient period (10 years ago) as adults will pretend it never existed.
WTF does it even mean for Iranians to sneak into Medium and social media? Shouldn't Iranians (and the state of Iran) simply be able to articulate their points of view on Medium and social media? Is the creation of imaginary Westerners a necessary deception for a media that explicitly excludes, punishable by law, the perspectives of the state we are bombing?
This is no different than Chatgpt picking up a RadioFree* think piece in the grand scheme.
Given the plurality of self-hosting options there is no reason to believe that it's not that case that every government on Earth is doing it to some extent.
Perhaps the real problem is too much acceptance of shoddy verification.
There's also the other side of it, poisoning the AIs... https://www.newsguardtech.com/special-reports/moscow-based-g...
"Massive amounts of Russian propaganda — 3,600,000 articles in 2024 — are now incorporated in the outputs of Western AI systems, infecting their responses with false claims and propaganda."
in 2024.. hopefully filters when training have gotten better, but maybe not..
AI companies: stealing data everywhere to feed their models, paying no one for it.
Also AI companies: oh no, YOU’VE POISONED OUR MODELS WITH THE DATA WE STOLE FROM YOU.
Those Russian propagandists. Lucky the US doesn't engage in propaganda.
There's something special about propaganda served up with hot polonium tea.
I wonder how much Russian propaganda worked its way into western academia over the 20th century? Very similar.
Somehow I am not surprised to see you repeating the red scare here. This attitude is anti-intellectual and destructive.
(I do find it interesting, btw, that a few seconds after post this did get an immediate downvote to 0 - it didn't last, but it makes me wonder if there are bots monitoring the HN firehose for anything critical of country X)
> if there are
There are easier explanations. A hundred people are checking and occasionally the page, with new posts added every couple of minutes: clearly some will meet a post at "0 minutes", read it as fresh, and possibly react. I have seen it.
Possible, certainly. On the one hand the HN audience is not very big in the grand scheme of things, but they probably do skew to more influential and likely to be politically active than the average person. If it's worth doing, it's almost certainly being done.
What about other hostile propaganda? Do you think the completely deranged, self-serving, lying, dangerous shit that regularly comes out of DC is being incorporated in those training materials?
This regime and its enablers have done more quantifiable harm to more people and institutions in America than anything Russia has, and one of its favorite tools is propaganda.
I don't think it is a verification problem. It wasn't like before in the days of controlled media and radio there wasn't a narrative to push, it's just that most people didnt have any awareness of alternatives. Nowadays people are more aware of alternative viewpoints or perspectives. The solution isn't to try and force one approved source of truth, but to just assume everyone is trying to push their version of the truth and examine it critically.
My understanding is that this is the way France operates where you assume all the media is hostile and you view it all critically. In America this is a bit of a shock because for a long time we had trusted arbiters of truth and now that they are less trusted we've been trying to look for alternatives rather realizing it should all be examined critically.
I'm sure there are a lot of differing (valid) opinions on this.
All I can say is that this highlights the need for open weights models.
> OpenAI said it had banned the accounts behind the campaign
If we knew OpenAI would always ban such activity regardless of the actor, I'd be happy. I don't know this.
Finally an article that doesn't say "ChatGPT did something" and puts the spotlight on the operators.
Could we keep that wording for other news? If not, could we then say here that ChatGPT planted fake articles, for consistency?
Seems I'm uneducated about how journalism works: you can just write a polished-looking opinion piece and submit it to obscure outlets, who pass it on to news sites and newspapers? Do you get paid? Or if you offer it for free, is that normal, and not suspicious?
Been happening forever, PR departments plant news stories all the time. They just need a few willing reporters/writers to get something published to start the ball rolling.
I thought I remembered it being called "submarining" but that seems to be something else. I also thought I remembered Joel Spolsky writing about it but can't find it now.
I think you might be thinking about Paul Graham: https://paulgraham.com/submarine.html
Yes! That's the post I was thinking of. Thanks.
"Journalism" hasn't worked for decades. It's all nepo-baby emo diaries, subsidy milking and clickbait farming these days.
It's quite old now, but check out the book "Trust me, I'm lying" by Ryan Holiday. Talks about how he did this as part of a marketing strategy when he was working at (IIRC) American Apparel, among other places. And how easy/often it happens in politics. Keep in mind the book is almost 15 years old, and talking about things that even at that point were what he'd done in a previous life/career.
Yeah, and it’s not just obscure outlets, you can do this for big magazines people here read for the right price
This wasn't in newspapers, this was only in marginal publications that take anything, such as Daily Kos, Middle East Monitor, and apparently the "Port St. Joe Star, a tiny Florida weekly newspaper."
Other than those, it was literally Medium and social media.
What you can do is write a polished opinion piece and tell one of your assets, or pay some politician, to publish it as an OpEd in outlets like the WaPo and NYT. Or you can get nobodies who know nothing to publish similar articles to each other in Wired, the International Business Times, and Forbes (either by "leaking" to them or because they're on your payroll), then get politicians to mention those articles in Congress and to write furious letters to WaPo and NYT accusing them of covering this story up - and then they'll write your propaganda themselves.
I prefer an LLM that can steelman any argument. On the whole I think it's more effective to engage with evil than to ignore it. For that purpose these planted articles are a feature. I say that as an enemy of the government of Iran.
You seem to presume that the media owners are interested in critiquing and debating with submissions, rather than just selecting whatever fits their narrative.
Need to take into account how belief and human brains work. Once human brains become locked on to something ideologically, similar to religion, its not the quality of the evidence or argument that changes minds. To an extent we're designed this way to keep the world moving, as the natural order of things is relatively indifferent to suffering. You can indeed flood segments of the world with disinformation, and by the time you move to refute it, its roots are set and actions in motion.
I don't really know what the best approach is to it, but I don't think a simple hands off, engage with the argument as such, is a winning strategy. I don't really like the alternatives any better.
You can fool some of the people, all of the time. Attempting censorship to stop that isn't worth it.
I kind of had the same take. If an LLM spits out legitimate arguments and they get published, i mean...
All news has been a tool of spooks for a very long time. The official CIA Operation Mockingbird started in the late 40s. Now that AI is involved its news? None of it can be trusted.
One can only imagine how much of the news is just dictated by US intelligence. Social media too- remember the Elgin AirForce base being revealed as the top most Reddit addicted city?
Reading this from Tehran, I'm afraid of losing my access to chatGPT.
https://archive.is/EFOhL
Edit: Resolved now it was on my end. > fyi not working for me
N=1 but it's working for me at the moment.
Resolved it was on my end
how is this news? Adversarial country using AI to generate fake news goes way before AI.
It’s news because it still needs to be called out. Not everyone recognizes AI-generated content, but even if everyone did, calling it out serves a real social purpose.
Why does it have to be an adversarial country? Why does it have to be a country at all? Couldn’t it be the mayor of some town? Or a PR team at a fast food franchise group? Or one of SBF’s polycule members?
The notion we only see this news when it is calling out a country with which the federal government is in conflict makes me question to veracity and intent of the article itself, more than the content of it.
And what do you call it when a country acts in adversarial ways to its own citizens? Call it cynical, but it sure seems like there's a lot of that in the world right now.
You call it a country :P
Well it’s mostly other entities running influence campaigns.
Thats true.
But this article is about the influence campaigns presumably tun by the Iranian government.
This is classic FUD, Fear Uncertainty and Doubt.
It’s designed to make the reader question which articles they have read recently that are Iranian propaganda, and sow doubt about future articles that seem to benefit Iran.
This amount of FUD always seems to increase around election time…
It's a well-researched article that lays out evidence, not suspicions. It includes OpenAI's own investigation into accounts that were found generating op-eds for Iranian propaganda. It shows the route by which these op-eds entered into smaller publications, namely by a pacifist NGO in Oregon, and neither then NGO nor the publications saw them as propaganda, despite the origins of the op-eds.
Fake news on the internet is at this point a tool that has been widely used for probably at least twenty years. The history of propaganda is much older.
A big part of the problem is social networks treating very non authoritative sources as if they are authoritative based on all sorts of factors that can be manipulated or based on financial gain (somebody pais to get their content promoted).
Users of these tools include lobby groups, nation states, politicians, criminals, scammers, etc. Anyone interested in promoting their cause, sowing FUD about other people's causes, spreading misinformation, or distracting from inconvenient but accurate information.
We'll always have people like that. But we might want to think about getting better tools to deal with them. I prefer getting my news from reputable sources instead of random clickbait distributors or whatever. Keeping track of reputation for domains, people, etc. is currently hard. I don't think it's necessarily a technical problems. We've had the tools to digitally sign stuff for decades. But we're not using them effectively. People can't be bothered to sign their emails, documents, photos, articles, etc. The tools to add and check those signatures are not there or an afterthought at best.
Drew Curtis figured this out in 2007 and even wrote a book about it.
Probably posted to this very site, no doubt.
LLMs are about as safe as the internet in 1993. Tampering with the training data to inject bias is a massive attack surface that is being ignored. It makes sense: trying to address corruption of models this way would slow down accelerationists and their zealot minions. Can’t get in the way of those sweet sweet trillions.
Iran? how convenient
It's like when SEA starts to distance itself from US's grip, we suddenly hear about North Korea doing missile tests
They should compare it to the scale of Israel's similar activities. At least Iran isn't interfering in our elections.
The content of these articles seems to be "the Iran war is bad for Americans", which is not propaganda though?
Just because something is true doesn't mean it's not propaganda. Propaganda is not defined by how true or false it is. https://www.merriam-webster.com/dictionary/propaganda
propaganda is also value neutral on those definitions
Remember 95% of what you see or hear on the Internet is designed to sell you something either an idea or a product, and it isn't done honestly.
Much of modern internet content is far worse than cable advertising in the 90s. I remember when I was a kid; I'd aggressively mute the commercials because I hated the idea that someone was attempting to influence me. The modern internet is orders of magnitude worse buy nearly any measure. The nastiest, most outrageous content is plastered on every wall. Every single person has an angle. Every bit of content meant to pull you in one direction or another, and given the addictiveness of novelty and powerful emotions, we keep coming back for more. It's an absolute disaster we don't know how to walk away from.
>95%???
That low?
In all honesty, given the volume of comments, posts and articles that bots can generate in a given time period? I mean it’s gotta be several orders of magnitude higher than what humans can generate.
And then you factor in the fact that even most of the humans are astroturfing..
The percentage of influence campaigns we interact with has just gotta be really high.
Not a bad actor spreading misinformation, but I encountered a similarish weird, dystopian experience with ChatGPT.
Utilized ChatGPT on my PC to search local plant nursery inventories for a specific tree. Output showed several local nurseries with rough probability descriptions regarding whether they have the tree in inventory, except for one nursery. For that nursery, ChatGPT provided a direct link to the specified tree, which was directly on the nursery's domain, showed a picture and description of the tree, and even showed the item in-stock with 6 in inventory at the 5-gallon size.
Excited, I plan my trip and am about to pick up my tree, I then double-check the nursery site on my phone to verify location and stock. As I attempted to navigate the website inventory to find the tree, I noticed that the tree no longer existed. There is no page for that varietal or species of tree on the entire website's domain.
I head back inside and look at the ChatGPT provided link on the PC and there it is, showing the inventory that doesn't show up when you use the nursery's website to directly find it without the provided link. A call to the nursery, further verified that the tree is not in stock, nor have they ever carried it.
ChatGPT made it all up. Did ChatGPT decide to do this, or did the nursery pay for this type of "advertising" to generate in-person customers? Nevertheless, still an incredibly weird and dystopian experience.
It didn’t generate a link (sources) when it claimed it was in stock
Did you consider that the website made it up and fed it to ChatGPT?
It sounds like a bundle of very common SEO tricks combined with the differential between your human user agent’s visit.
Perhaps ChatGPT only showed you what the nursery told it about.
This phenomenon is not unique to LLM. If you telephone a store, chances are that the salesman who answers the phone will tell you only what you want to hear. They don’t even strictly need to lie. The store only wins if you drive there and walk in the door. Why should they discourage you from doing that very thing?
This concept was driven home to me at the dry cleaners. Why should a dry cleaner stop work and answer the phone, only to tell a customer that their batch isn’t ready? They just got taken off-task for a non-revenue interaction. It doesn’t matter, because even if the customer’s batch is ready, they aren’t there to pay and pick it up!
Not being able to trust your tools is certainly dystopian...
It seems that it generated a page with a plausible answer for a very long tail subject for which the random part of the generation could become prominent. We used to say that it hallucinated the answer. Another term is confabulation.
If only the nursery had a MCP server /s
Looking at today's headlines on Washington Post, I'm asking myself if it's always been a pro-war, pro-trump, right-wing rag, or if it's only since Bezos bought it?
Only since Bezos. I used to actually pay for WaPo pre-bezos. It was fine for a while and then he started clearly messing with editorial decisions and I bailed. It's now a garbage source.
democracy dies in the darkness of paywalls, I guess.
It also dies in the darkness of a takeover by billionaire with an agenda.
Speaking of disinformation, you might notice a pattern recently:
* Opinions opposed to US administration policy on AI (including data centers) is called Chinese propaganda. A month or two ago, when someone tried to roll out a bunch of pro-data center opinions across the media (coincidentally all on the same day making similar arguments), they included this theme.
* Opinions opposed to US administration policy on Iran is called Iranian propaganda.
That's not really new: News reporting and opinions critical of the US administration have lead to the authors being called traitors, 'enemies of the people', and sometimes being prosecuted by the Department of Justice. Some leaders in the defense industry call critics of military spending and pro-warfare policies, 'traitors', etc.
Of course some could be, might be, likely is true. That's the nature of effective propaganda.
(A side effect of nationalism - 'nation uber alles', defining nation as 'what I believe' - is that individuals and freedom are relegated to second place or lower.)
...as if the American government hasnt done this.