The Lizardman Constant: What It Is and Why You're Misunderstanding It
fun with public health surveys
Public surveys are fun! You learn lots of important things about the world and the people within it. For instance, we know that 1 in 50 people don’t know police exist, 1 in 20 atheists believe in God, 1% of Israeli ultra-Orthodox either eat pork sometimes or “don’t know” if they do, and 1 in 25 people have been decapitated. Hm, maybe there are some problems with those ones.
The general phenomenon that some proportion of survey responses are bizarre is pretty well-known. A famous name for it is the Lizardman Constant, from a blog post that used to have the excellent name “Noisy Poll Results and Reptilian Muslim Climatologists from Mars”. Unfortunately, it was later renamed to “Lizardman Constant is 4%”. I now get to regularly see people saying things like “well, 6% of respondents answered yes to question #33, which is just above lizardman, so it’s probably 2%”. This is not how population data works! People give weird survey responses for a lot of reasons, and this will comprise a totally different (low) percentage depending what your population and questions are.
What are some of these reasons?
Some inherent baseline is “Whoops, I clicked the wrong option”. Unfortunately, this doesn’t seem to produce a consistent threshold you can subtract from your results.
Pew Research is the gold standard for this. They do great surveys and publish everything, so you can subdivide exactly how many Southern Baptists are certain God doesn’t exist. This is “less than 1%”, i.e. negligible. Of course, about 0.4% of Americans report being somewhere in the New Age/Modern Pagan cluster, and you’ve probably met a few of those.1 If 1 in 250 Southern Baptist respondents said they were certain God doesn’t exist, this is something different to 1 in 250 total respondents saying they were in the Greater 1960s Sphere. Maybe it wasn’t 1 in 250 — maybe it was 1 in 500, or 1 in 1000, or no one at all — but we don’t know.
There are rarer religions than this that we know have practitioners. 1.6% of respondents picked Mormon; functionally all of them said they were members of the Church of Jesus Christ of Latter-day Saints. Not all Mormons are part of the mainstream LDS Church. Mainstream Mormonism hasn’t practiced polygamy for longer than anyone on Earth has been alive, but the splinter sects that do are prominent enough for a lot of cultural persistence. Mormon fundamentalist groups are small, but obviously exist.
Of course, some people mis-answered that they were Pagans or Mormon fundamentalists. But you can’t exactly subtract that from the results, can’t you?
Similarly, we have the aforementioned survey on pork-eating habits amongst Haredi Jews in Israel. This is part of a larger survey on the religious practices of Israeli Jews, which subdivides by four levels of religious affiliation and includes a huge diversity of questions. There is no consistent, obvious trend in “which weird subdivisions have nonzero answers”. Notably, there were hard zero answers for “Haredim who don’t keep kosher at home” (including any uncertainty about whether you do, or any exceptions to a general rule), yet 1% would sometimes eat pork or be unsure if they do. There’s evidently something wrong there!
So, the first thing we’ve learned: there’s no baseline of obviously mistaken answers we can subtract across all surveys. What’s the next problem down the line?
Surveys are weird. In particular, forced-choice surveys are weird. Do you know exactly where your political beliefs lie on a scale from Most Liberal to Most Conservative? Yeah, me neither. (“I need a y-axis. No, a z-axis.”)
Because surveys are an unnatural environment, peoples’ responses on them can’t be linked unambiguously to their real-world experience. Responses to any survey should be read as “what answer has the closest vibes”. “Vibes” are broadly construed, may or may not resemble reality, and definitely don’t need to be consistent with each other.
This is why I don’t blink at “1 in 25 people say the lizard aliens are real”. Yeah, that has the right vibes, I can see it.
Vibes-based answering can produce wackily high wrong-responses. 6% of people think they could beat a grizzly bear in an unarmed fight. If you actually offered a few thousand people the opportunity to fight a grizzly bear one-on-one bare-handed, I think well below 1 in 16 of them would take you up on it. It’s the vibe of the question. “Do you think you’re the kind of guy who could do this?”
As the questions disconnect further from reality, the answers get more ridiculous. This is a survey of Americans, i.e. people most familiar with North American fauna. 17% of male respondents thought they could win a fistfight with a kangaroo. “Yeah, it’s a herbivore, right? What kind of guy would I be if I couldn’t beat a herbivore?”
Kangaroos look like this:
You will not win a fistfight with a kangaroo. (Let’s not even talk about how many men thought they could beat chimpanzees.)
When questions are heavily “vibes-based”, weird responses mark reactions to this, not literal-statements-of-actual-reality. Notably, this is also true of not-weird-seeming responses. Do you Disagree or Strongly Disagree with a statement? Eh, screw it, it’s all vibes.
You can see great examples of this back in the Pew Religious Landscapes survey. About 7% of people say both that they have no particular religion and that religion is important to them — roughly as common as atheists and agnostics combined. This is unusual when you lay it out like that, but makes perfect sense as an expression of unaffiliated spirituality. Similarly, the largest share of agnostics express confidently that they don’t believe in God; only 13% of agnostics say they don’t know if they believe in God or not. This has some obvious definitional issues, but agnostic identity in practice seems very atheist-of-centre, so “the largest share of agnostic-identifying people are uncommitted atheists” checks out.
Huge shares of respondents claim they routinely experience transcendental wonder, deep senses of spiritual peace, etc. In the real world, most people don’t seem to experience transcendental wonder several times a week. I’m sure I’m at least 95th percentile for transcendental wonder, and I’m not convinced it makes it there. But hey, are you the type of guy who says you do on surveys?
All of this assumes good faith, but not everyone approaches surveys in good faith.
Some subset of survey-takers are “mischievous responders” — people making things up to troll the researchers. In adult surveys, this is a pretty small subset of respondents. In youth public health surveys, it’s a massive problem.
The proportion of teenagers who will jokingly say on a public health survey that they’re gay/trans/disabled/parents/etc is large enough to rival the proportion of teenagers who are actually any of those things. This seriously skews our understanding of them. Cimpian et al. (2018) found that excluding the most suspicious subset of responses from a teenage health survey massively decreased the apparent correlations of being gay with drug use, as well as several other health outcomes.2 You might similarly be familiar with reports that claim gay kids are more likely to be teen parents — a suggestion with obvious logistical issues, but taken seriously by some headlines.
(I like the specification of top/bottom 2.5% here, because it implies a level of exaggeration much more subtle than most teenagers probably would. In reality, that’s all kids saying they’re like 8 feet tall, right? I dig the idea of some 15 year old going “damn, I gotta pretend my height is exactly at the 98th percentile”.)
Even rarer things have even worse response patterns. The famous case of this is Fan et al. (2006) regarding artificial limbs. In a large (90k participants) public health survey, 15k respondents were followed up through in-person interviews with themselves and their parents — a much harder scenario to bullshit in.
99% of participants who claimed they were missing a limb were lying!
In fact, more teenagers who said they didn’t use artificial limbs were actually using them (ten no/missing answers were really missing a limb, compared to only two yes answers). The researchers didn’t interpret these respondents as lying — the in-person follow-ups were a year later, and maybe all ten of them were injured in the past year — but the largest share of people under 18 who are missing limbs were born without them (McLarney et al., 2021),3 so they probably didn’t all fall under cars in a 12-month period. There is no moral to this story.
Supposedly, if you just ask teenagers at the end of a survey whether they lied or not, 12% will say they did. You’d better hope that wasn’t the only question they lied on before you throw out the data, though.
Some questions, and some surveys, are more likely to inspire dishonesty than others.
This is the basic problem with extrapolating a 1/25 wrong-answer rate from “do you believe reptilian aliens rule the Earth” and “have you ever been decapitated”. Those questions are obviously ridiculous. “What do you mean, the pollsters are so dumb they’re asking if I’ve had my head cut off? Fuck it, we’ll do it live.”
This introduces a tricky problem with “attention”-style questions. If you add obviously weird questions to catch out people who are responding dishonestly, you might also catch people who wouldn’t otherwise do so. We know (sic) that there are twice as many people who’ve been decapitated as who have never heard of China, and four times as many as the proportion of Ultra-Orthodox Jews who eat pork, so this question obviously inspires dishonest responses in more people than usual.
In extreme cases, this is a problem on the whole survey level. Public health surveys for adolescents are a previously-reviewed example. Another is internet surveys for culturally/emotionally/politically charged topics. Consider the widely-reported survey finding that covid vaccine hesitancy was highest amongst people with doctorates. It turned out that while 3% of respondents claimed to have PhDs, 28% of those who picked “Prefer to self-describe” for their gender did, and ~all of these were meme “attack helicopter” answers. It also turned out many of them claimed to be Hispanic and aged 75 or older. “Once you remove elderly Hispanic attack helicopters, the effect largely disappears.”
I strongly suspect that “other/nonbinary” gender options have this problem across surveys, and in particular that a lot of large and widely-reported LGBT health surveys have high mischievous responding, but it doesn’t seem that anyone’s looked into this.
Unfortunately, people want surveys on culturally/emotionally/politically charged topics! These are precisely the subjects it’s important we figure out. Whether we can do so in any sane or coherent way is another question.
I have a study design in my head that’d be able to, as a side consequence, test a lot of mischievous/erroneous responding. This is the velocardiofacial syndrome survey discussed before — get $number teens with VCFS (a genetic disorder that causes a high lifetime psychosis rate), $number control teens of varying intellectual ability (people with VCFS have an average IQ around 75, and most studies investigating similar things haven’t controlled for this), and $number teens with a psychotic first-degree relative, test various personality traits (schizotypy, Five-Factor Model). For logistical reasons, this needs to have inclusion criteria of “has never been psychotic” and “isn’t autistic” (would confound schizotypy questions). It could also use a couple other inclusion criteria to ensure understanding of the questions, e.g. absolutely double-checking that someone is able to read English fluently.
As it happens, if the extensively double-checked entry criteria for your study are also potential mischievous responses, you can just check who’s going to answer unreliably! “Do you have an autism spectrum disorder?” should get a 0% yes rate;4 inasmuch as it gets higher rates, these responses are dismissable. Same for “are you schizophrenic”, “are you blind”, etc. But that doesn’t mean kicking them out of the study — if you follow all their responses in a separate Confirmed Erroneous Responders bucket, you can gather a lot about the patterns of such responses, the kind of people (from parent-confirmed demographics and information) who make them, etc.
This can’t be done in large population surveys, which precisely by their nature can’t pre-vet participants. I think the knowledge of e.g. what patterns suggest mischievous or erroneous responding would be super-useful, though.
(again, if you’re in a position to assist with such a project, I am seeking that)
Now that we’ve reviewed all these common ways answers can be wrong, we can at least confidently dismiss small wrong-answer percentages as equally meaningless, right? Ha ha no, you wish.
Let’s go back to “2% of people don’t know police exist”. There is obviously something wrong with this estimate. Similarly, 1 in 50 people (each) have never heard of the military, Amazon, the Department of Justice, China, or Russia. Again, probably something wrong here.
The “never heard of” responses follow gradients, though, and there are interesting patterns to these gradients. 2% of respondents have never heard of China or Russia, but 5% have never heard of Israel, and 11% never heard of Palestine (this survey is from 2021). All of these are obviously huge overestimates, but the patterns are correct! China is the largest country on Earth, there are Chinese-inspired restaurants in the smallest of Anglosphere towns, the media talks about China constantly, and most people will meet a Chinese person at some point.5 Israel is a much smaller country that, pre-recent-events, is of fairly distant geopolitical relevance to most people. The numbers are order-of-magnitude overestimates, but the discrepancy is right.
Similarly, 1% of respondents have ‘never heard of’ Facebook, and 4% have ‘never heard of’ Twitter. (So, better-known than China and Israel respectively.) Again, wrong proportions, right directions. Twitter has always been a niche site with outsized chattering-class influence. Very few people would be unaware of Facebook, but having a hazier sense of Twitter must be more common.
8% reported never having heard of the European Union, which again is a presumable overestimate, but one that reflects an accurate “more people have heard of the FBI or military than of the EU”. It’s also about the same percent as had “very favourable” views of Putin (9%), which is funny.
This ties into the matter of vibes-based responding. Surveys are an unnatural environment and don’t capture your real views. This is why adding “none of the above” options instantly ruins your data — everyone picks it for their Special Snowflake Situation instead of picking a forced-choice. That doesn’t mean the forced-choice options get at reality, though, just a measurable reality-like thing.
The correct vibes answers can be bizarre or implausible or contradictory. I know I’ve given survey response patterns that must’ve looked like junk data. It was the best portrait of reality I could construct from the options.
Curran & Hauser (2019) is an amazing study of this. They surveyed undergrads who gave implausible answers to a set of questions as to why they chose them. Responses included:
(“I can teleport across time and space”) “Well, time passes, and I can move places, so that’s sort of true”
(“I am paid biweekly by leprechauns”) “I’m paid biweekly, just not by leprechauns”
(“I work 28 hours in a typical workday”) “It feels like that sometimes”
(“I can run two miles in two minutes”) “It didn’t say ‘can you do it with your feet’, I can do it in my mind”
(“All my friends say I would make a great poodle”) “They say I’d make a great koala”
People: they’re complicated!
Then, for additional fun, some of the “implausible questions” were clearly taken from a schizotypy scale. People: they have weirder experiences than you think!
Bizarre response patterns don’t represent a clear baseline of Unambiguously Wrong Answers, but a complex process of “people don’t take surveys at face value, because they inherently have hard map-territory disconnects”. They don’t form a consistent flat percentage you can subtract; they might be <1%, they might approach 10%. Responses that don’t seem obviously bizarre are being answered the same way, with the same heavy caveats. They’re also sometimes just lies, but this itself is complex.
The “lizardman constant” is better thought of as a phenomenon. There’s no flat wrong-answer constant, but a general phenomenon of complex responses to simple-sounding questions. The world has more diversity of experience than you think, and it runs stubbornly, impossibly up against attempts to categorize it.
I think this is probably an underestimate, in that people with esoteric beliefs answer very different things on surveys. The broad “spiritual weirdo” cluster is most likely split between Pagan/New Age, spiritual-but-not-religious, religious nones, syncretists who answer with their mainstream faith, and white Buddhists. Also, to be clear, “Paganism as subset of New Age” is Pew’s division, not mine.
The drug use drop is large for both gay men and lesbians, but some of the others are more complex. Removing supposed-gay-male respondents that track as mischievous significantly decreases the number who say they’ve been raped?raped someone?it’s not very clear, but I think it’s raping someone, but the same isn’t true for lesbians. The same male-only large drop is true of suicide attempts (but not ideation, which remains consistently high) and, to a lesser degree, number of sexual partners.
There’s a bit of equivocation here, sorry. This is a comprehensive survey (all children and adolescents in the United States who had private insurance medical codes related to missing limbs between 2009-2015), but it’s only lower limbs and the whole age range. Teenagers are presumably more likely to be missing limbs due to trauma compared to children. Public insurance (i.e. Medicaid) also wasn’t included, and kids covered by that are disproportionately seen at hospitals. Having said that, over half the respondents were adolescent/preadolescent, and congenital causes had a huge lead.
Yes, the base rate would be more than 0%, because diagnosis is wildly imperfect. We’re assuming a “say no if you suspect it or are self-diagnosed” clause. In practice, I expect vibes-based answering means people routinely ignore these clauses, but it’s the best we can do.
Is the last of these actually true of Americans? I found myself bet-hedging a lot about what to say for the last clause, because a much larger proportion of the Australian than American population is of Chinese descent, and Asian-Americans are pretty geographically clustered in ways that make extrapolating from California even worse than usual. It’s hard to think it’s not true of “people who visit cities at some point in their lives”, but that’s an Australian view of cities.
wow this made me remember something I hadn't thought about in years. My dad used to give me Gallup surveys when they came in the mail when I was like eight (sorry in retrospect I realize this is bad). Anyway one time he looked over my responses and his face kinda softened and he was like "awww Annie I didn't know you were so terrified of terrorism all the time" and then we went to ToysRUs and I got a Barbie. Such a mid-2000s thing to happen lol. also, great newsletter as always!
> Amazon, the Department of Justice
I don't think 2% of people not having heard of either of these sounds particularly unlikely. I can well believe 2% of people (elderly, isolated, mild dementia) have never heard of Amazon. And I don't think I could confidently say I'd heard of the Department of Justice. Granted, I'm not from the US, but plenty of people have virtually no understanding of the government even of their own countries.
Police is a stretch, though, I admit.