polls, surveys, quizzes, and questionnaires

[asakiyume]

Sometimes little online quizzes and things can be fun--things that tell you which character in XX show/book you are, or whatever. But as you've probably all experienced taking those quizzes, they reveal as much or more about the assumptions of the quiz creator as they do about you.

That's one of my huge--sometimes insurmountable--problems with health questionnaires or political polls, too. (With political polls, another huge problem I have is who the hell is actually answering them. I *never* respond to political polls--largely because of problems related to the questions, but also because I don't answer spam phone calls etc.--so what person has the time and inclination to answer them? How biased are all polls, when they only draw on the pool of people willing to respond to polls?)

One basic thing that all quizzes, polls, and questionnaires often do wrong is that when they offer you responses, they also supply reasons for the response. For example:

Q. Do you pick up roadside litter?

A1. Yes, always; I can't stand that jerks are always leaving their trash around.
A2. Yes, sometimes, if I'm not in a hurry
A3. No, never--why should I pick up after other people?

All these answers have rationales that may not be the respondents' rationale at all. For instance, maybe you never pick up trash because you're immunocompromised and don't want to get sick. Or maybe you always pick up trash because you're interested in what things people discard. Or maybe you sometimes do, but it's not so much whether you're in a hurry or not but whether you have a free hand. And so on. (Never mind that the answers don't allow for certain meaningful other answers, such as "Yes, but only cans and bottles that I can collect the deposit on.")

If the rationale in an answer isn't your rationale, then you're forced to either choose that answer, even though it misrepresents your thinking, or not answer at all.

For that reason, I prefer questions that don't put rationales in the response choices .... But of course, the quiz creators would still be assuming the rationales, only their assumptions are hidden from you.

All of this is to say, always, always be skeptical of responses to questionnaires and polls. You don't know what kind of crap thinking went into their creation, how they were worded, and so on.

Comments

[lilysea]

Indeed.

"No, I don't pick up rubbish - bending over aggravates my back/hip pain for days" is not an option...

Around 1996/1997/1998/1999 I once had a market researcher come to my door with a clipboard and ask how many hours of TV I watched - zero, I said.

They flat out refused to believe me - partly because zero was not an option on their form.

Finally I had to say "Look, MY HOUSEMATE MOVED OUT AND TOOK THE TV. THERE IS LITERALLY NO TV IN THE HOUSE. THEREFORE IT IS IMPOSSIBLE FOR ME TO WATCH MORE THAN ZERO HOURS OF TV"

[shewhomust]

Yes to this! I recently saw a meme hereabouts which included the question 'Have you ever lived in a house without a dishwasher?' (I have never lived in a house with a dishwasher.)

My pet hate is the survey that forces me to select a neutral 'don't know / don't care' option to express 'I have SO MANY opinions about this...'

[zyzyly]

I tend to avoid those quizzes, as I am always thinking there is someone who wants that information for something nefarious.

[sartorias]

Yeah, this is why I rarely even look at them, much less answer them. Though I used to read them to see what the thinking was behind the questions, but even that falls into predictable patterns. That said, if someone I trust puts up a poll,, or if there is one that makes me laugh, I'm in.

[anna_wing]

I only ever do the ones that are blatantly frivolous, like "Which character in a major recent media property are you?" (someone called Littlefinger from "Game of Thrones" which I don't recall if I bothered to look up, since I have never watched the TV series, and hurled the first book aside in irritation at how dull it was...it was at least a nice, memorable name, which is why I remember it).

Personality-related questionnaires, whether serious or frivolous, always fall down by failing to make the distinction between personality traits and actual behaviour. They are also seem to be always designed for and by Americans.

[mrissa]

I have often said that on the rare occasions when I do take a personality quiz, it's to see how reasonable the quiz is, not to see what it says about me. If there's a "What Race Of Middle Earth Are You" quiz that does not give the result that I am of the Rohirrim, it's a bad quiz. But there are in fact plenty of bad quizzes.

[wayfaringwordhack]

You don't know what kind of crap thinking went into their creation, how they were worded, and so on.

Oh, yes. This. It is so easy to twist the most seemingly benign info into a Thing It Is Not to support the Thing They Want It To Be.

o.O

[hyba]

Great points! I remember that in uni they made a huge point of making sure we understood how to create surveys and undertake interviews so that when we designed them, our questions didn't lead respondents to a specific response or rationale. That would ultimately make at least some of your survey results inadmissible. For example, in the example you gave, nobody would pick the last choice because the rationale is phrased in such a way that it feels damaging to one's reputation - almost as though they aren't community-minded (which, depending on the society the respondent lives in, could be a huge no-no). So, naturally, the survey designer has led them away from that response.

So at the end of the day, what you had to do if you wanted to let everyone give them honest and detailed reasoning - and have more accurate results - is to create other questions that people can actually respond to in free-form. But that means more work afterwards, and you'd have to sift through all the responses and create categories for them to be able to analyze them statistically, and a lot of people are just too lazy to do that, I suppose.