And here I was thinking this was proved beyond any reasonable doubt when 4chan users spoofed the Time Magazine person of the year poll of 2009 to spell out "MARBLECAKE, ALSO THE GAME" by the initials of the top winners:
This is evidence, but it would be a mistake to think that high profile online contests would have the same dynamics as low profile online polling. 4chan is characterized by an insane level of effort on the specific jokes that they care about. But they don't care about a random poll nearly as much as they care about big stunts like this.
Online polls being easy to manipulate makes it so weird to me that the European Central Bank made an online poll on what the new Euro bank note design should be. (Participants did not have to identify themselves in any way.) I wonder if the results will at all be taken into consideration in the following decision-making process.
In that case there was heavy pre-selection und strict rules for submissions even before that. So they were probably fine with any of the presented choices winning and the "poll" is really just PR.
In fairness I think a lot of people (me included) genuinely thought that would be an entertaining name for the ship, so the polling result wasn't necessarily wrong per se.
Just a throwaway kinda funny story that's obliquely related to this. I'm not a pollster, I'm a developer. I was tasked maybe 10 years ago with creating a feedback survey that a hospitality company wanted to email to customers after their visit. It was a mix of yes/no, rate 1-5, and free writing questions, all optional.
The responses got stored in a database, as well as being emailed raw to the relevant managers. The numerical responses could be quantified and tracked over time, but there was pretty much no standard of measurement for the hundreds of thousands or millions of free writing boxes over the years.
A year ago I set up a small locally run classifier model to back test and "score" all that stuff, and to try to reveal patterns that might have been hidden.
The classifier did turn up a bunch of sets of complaints that had been overlooked by upper management. But even more so, it flooded the company with false positives of negative reviews.
It turns out that a lot of positive reviews include false-negative idiomatic expressions like "killer service" or "it beat the shit out of staying at...", which the classifier considered extremely negative.
Just a warning to anyone who thinks LLMs will help clarify the user's intent... they won't. This really makes me ponder the value of putting faith into the voodoo of the LLM's probability metrics, a la Jev. I think you may just be betting on a derivative of a fundamentally broken system, where trusting some early underlying abilities leads you to lean on a confidently presented but ultimate unsupported probability number you want to bet on, but which has no mathematical relationship to any real probability.
The bit about 10% of respondents claiming they had visited the International Space Station was funny. Reminded me of this nugget[0], which for people unfamiliar with the subject doesn't mean that 20% of Jews believe in Jesus... it means that 20% of the respondents to the poll were either deranged or confused, and self-identity as Jewish for no clear reason. It's analogous to "1 in 5 vegans say they eat meat."
Righht, which is about as accurate as checking the box that you're Native American because your grandpa said you might be 1/16th Cherokee.
Assuming an identity doesn't make it your culture, right? It says more about the person who assumes it. Messianic Jews are a form of bonkers identitarians who existed even before the current fad.
No, there's no quotient of Jewish people who would take the question of belief in Christ to mean "do you believe Jesus was an historical person who walked around Samaria in sandals". Pretty much all Jews agree that Jesus was a real renegade rabbi who wore shoddy footwear. The definition of being Jewish is that you don't believe he was the Messiah, because the Messiah hasn't arrived yet. Anyone who clicks the buttons that they're Jewish and that they believe in the divinity of Jesus is clearly a throwaway on the survey.
Movies and TV tell me everyone in Judea and Samaria wore sandals back then. Why single out one guy? Or was Jesus's pair particularly scuffed, ratty, and worn-out?
Something like 290 people have visited the ISS. If they all answered the survey and if the survey size was around 2900, then the 10% figure would be correct.
I cannot find the paper again, but a small group of women giving birth in the US healthcare system do self report as virgins. There is not protecting for misunderstandings or malicious intent, even in person.
From time to time we've battled multiple ideas around polling and tech within my friend circles, because there really isn't "reliable" way of knowing "what people want" when it comes to government's decision makings. It's such a hard "reach and incentives" problem, that asking for an opinion, might change the opinion itself as well, depended on the questioning. That being said, it would be cool to see real opinions of people on general matters, rather than heavily sampled versions of it.
I have no sympathy for polls. Y'all sending SMS unsolicited is in violation of the TCPA, and I'm not going to respond to an organization that has so little respect for me.
You need to though. If you only do phone call polls you're only going to poll people with (landline?) phones who answer (random?) calls and have the time/enthusiasm to respond when you call. That heavily skews demographics.
A lot of people will only do online polling, and it's by far the easiest way to cover a lot of demographics (need to supplement with other methods as well if you don't want a poll of "internet users" - but that is a far larger and varied group than most other methods would produce)
Getting a different error is great! Then if you do an online poll and a phone poll, you can combine the results to get a composite poll that's more accurate than either method alone.
Both oversample people that want to take polls, that's the problem.
In fact, online polls are worse because you now have to make sure the "people" answering are actually people and are actually able to influence whatever polling data you're trying to gather.
They've gone from gathering a dataset that's been becoming less reliable to one that is even less reliable.
Ah this is a good point; you've caught my mistake - I only saw "online" and "poll".
Edit: I am a poor survey respondent to non-opt-in polls, and do not opt-in to polls where possible.
--> Claude informs me ("why are there bogus respondents to polls you have to opt-into anyway?") that most are there because surveys pay a little money; so it's botted too.
Also by "straight-liners," as we call them: people who just click items in a straight line to finish asap. And then there are the nonsense respondents. Don't trust poll data.
Betting markets are even less representative. People who trade on politics are usually rich conservative men with a risk taking mindset and more likely to be active crypto users or work in finance. It's a totally different user base from normal polls.
The point of markets isn't to recruit a diverse user base, it's to create an incentive structure that punishes bad predictions and rewards good predictions.
And here I was thinking this was proved beyond any reasonable doubt when 4chan users spoofed the Time Magazine person of the year poll of 2009 to spell out "MARBLECAKE, ALSO THE GAME" by the initials of the top winners:
https://techcrunch.com/2009/04/27/time-magazine-throws-up-it...
Or when Kim Jong Un won in 2012, spelling out "KJUGASCHAMBER":
https://observer.com/2012/12/4chan-successfully-votes-kim-jo...
You're half right, half not.
This is evidence, but it would be a mistake to think that high profile online contests would have the same dynamics as low profile online polling. 4chan is characterized by an insane level of effort on the specific jokes that they care about. But they don't care about a random poll nearly as much as they care about big stunts like this.
Of course, but a low profile poll also needs less effort to manipulate, so it scales down for both ends.
Online polls being easy to manipulate makes it so weird to me that the European Central Bank made an online poll on what the new Euro bank note design should be. (Participants did not have to identify themselves in any way.) I wonder if the results will at all be taken into consideration in the following decision-making process.
https://www.ecb.europa.eu/press/pr/date/2026/html/ecb.pr2609...
In that case there was heavy pre-selection und strict rules for submissions even before that. So they were probably fine with any of the presented choices winning and the "poll" is really just PR.
Or Boaty McBoatface winning the name of a research ship (given to a submarine on the ship).
In fairness I think a lot of people (me included) genuinely thought that would be an entertaining name for the ship, so the polling result wasn't necessarily wrong per se.
Fair, though it does show how things like that can skew polling results.
i.e. engagement bait
At the time, we ended up with a conference room with the same name, after an office vote.
Those were not bogus respondents, just silly ones.
Just a throwaway kinda funny story that's obliquely related to this. I'm not a pollster, I'm a developer. I was tasked maybe 10 years ago with creating a feedback survey that a hospitality company wanted to email to customers after their visit. It was a mix of yes/no, rate 1-5, and free writing questions, all optional.
The responses got stored in a database, as well as being emailed raw to the relevant managers. The numerical responses could be quantified and tracked over time, but there was pretty much no standard of measurement for the hundreds of thousands or millions of free writing boxes over the years.
A year ago I set up a small locally run classifier model to back test and "score" all that stuff, and to try to reveal patterns that might have been hidden.
The classifier did turn up a bunch of sets of complaints that had been overlooked by upper management. But even more so, it flooded the company with false positives of negative reviews.
It turns out that a lot of positive reviews include false-negative idiomatic expressions like "killer service" or "it beat the shit out of staying at...", which the classifier considered extremely negative.
Just a warning to anyone who thinks LLMs will help clarify the user's intent... they won't. This really makes me ponder the value of putting faith into the voodoo of the LLM's probability metrics, a la Jev. I think you may just be betting on a derivative of a fundamentally broken system, where trusting some early underlying abilities leads you to lean on a confidently presented but ultimate unsupported probability number you want to bet on, but which has no mathematical relationship to any real probability.
The bit about 10% of respondents claiming they had visited the International Space Station was funny. Reminded me of this nugget[0], which for people unfamiliar with the subject doesn't mean that 20% of Jews believe in Jesus... it means that 20% of the respondents to the poll were either deranged or confused, and self-identity as Jewish for no clear reason. It's analogous to "1 in 5 vegans say they eat meat."
https://www.timesofisrael.com/jews-for-jesus-poll-15-of-jewi...
Could be a combination of messianic Jews and some Jewish people assuming that the question is about the historical yeshua
Righht, which is about as accurate as checking the box that you're Native American because your grandpa said you might be 1/16th Cherokee.
Assuming an identity doesn't make it your culture, right? It says more about the person who assumes it. Messianic Jews are a form of bonkers identitarians who existed even before the current fad.
No, there's no quotient of Jewish people who would take the question of belief in Christ to mean "do you believe Jesus was an historical person who walked around Samaria in sandals". Pretty much all Jews agree that Jesus was a real renegade rabbi who wore shoddy footwear. The definition of being Jewish is that you don't believe he was the Messiah, because the Messiah hasn't arrived yet. Anyone who clicks the buttons that they're Jewish and that they believe in the divinity of Jesus is clearly a throwaway on the survey.
> real renegade rabbi who wore shoddy footwear
Movies and TV tell me everyone in Judea and Samaria wore sandals back then. Why single out one guy? Or was Jesus's pair particularly scuffed, ratty, and worn-out?
Something like 290 people have visited the ISS. If they all answered the survey and if the survey size was around 2900, then the 10% figure would be correct.
See also: Lizardman's Constant is 4%.
https://slatestarcodex.com/2013/04/12/noisy-poll-results-and...
Having dealt with this before we always put trap questions in polls. Filtered 3-7% of respondents.
[flagged]
I cannot find the paper again, but a small group of women giving birth in the US healthcare system do self report as virgins. There is not protecting for misunderstandings or malicious intent, even in person.
From time to time we've battled multiple ideas around polling and tech within my friend circles, because there really isn't "reliable" way of knowing "what people want" when it comes to government's decision makings. It's such a hard "reach and incentives" problem, that asking for an opinion, might change the opinion itself as well, depended on the questioning. That being said, it would be cool to see real opinions of people on general matters, rather than heavily sampled versions of it.
I have no sympathy for polls. Y'all sending SMS unsolicited is in violation of the TCPA, and I'm not going to respond to an organization that has so little respect for me.
The easy fix is to not do online polling.
You need to though. If you only do phone call polls you're only going to poll people with (landline?) phones who answer (random?) calls and have the time/enthusiasm to respond when you call. That heavily skews demographics.
A lot of people will only do online polling, and it's by far the easiest way to cover a lot of demographics (need to supplement with other methods as well if you don't want a poll of "internet users" - but that is a far larger and varied group than most other methods would produce)
Actual visiting. Has been done. Still seems to be the preferred method in parts of the world with unreliable internet.
That's like punching the ground because it hurts to punch the wall.
Online polling isn't an improvement, it's just a different error.
Getting a different error is great! Then if you do an online poll and a phone poll, you can combine the results to get a composite poll that's more accurate than either method alone.
Both oversample people that want to take polls, that's the problem.
In fact, online polls are worse because you now have to make sure the "people" answering are actually people and are actually able to influence whatever polling data you're trying to gather.
They've gone from gathering a dataset that's been becoming less reliable to one that is even less reliable.
[dead]
How about just not doing online opt-in polls
Nobody enjoys that experience; I personally am a "bogus" respondent.
Why do you opt in to the poll then?
Ah this is a good point; you've caught my mistake - I only saw "online" and "poll".
Edit: I am a poor survey respondent to non-opt-in polls, and do not opt-in to polls where possible.
--> Claude informs me ("why are there bogus respondents to polls you have to opt-into anyway?") that most are there because surveys pay a little money; so it's botted too.
The lure of a few bucks for minimal effort always attracts fraud. Our own survey data is consistently tainted by bots and click-farmers.
Also by "straight-liners," as we call them: people who just click items in a straight line to finish asap. And then there are the nonsense respondents. Don't trust poll data.
Polls seem completely useless over the past decade or so. I think betting markets are a better indicator of true sentiment now.
Betting markets are even less representative. People who trade on politics are usually rich conservative men with a risk taking mindset and more likely to be active crypto users or work in finance. It's a totally different user base from normal polls.
The point of markets isn't to recruit a diverse user base, it's to create an incentive structure that punishes bad predictions and rewards good predictions.
Survey fatigue is real. It seems that every transaction, larger or small, involves a follow up survey now.
[flagged]
[dead]