asakiyume: (miroku)
asakiyume ([personal profile] asakiyume) wrote2020-11-11 10:05 am

polls, surveys, quizzes, and questionnaires

Sometimes little online quizzes and things can be fun--things that tell you which character in XX show/book you are, or whatever. But as you've probably all experienced taking those quizzes, they reveal as much or more about the assumptions of the quiz creator as they do about you.

That's one of my huge--sometimes insurmountable--problems with health questionnaires or political polls, too. (With political polls, another huge problem I have is who the hell is actually answering them. I *never* respond to political polls--largely because of problems related to the questions, but also because I don't answer spam phone calls etc.--so what person has the time and inclination to answer them? How biased are all polls, when they only draw on the pool of people willing to respond to polls?)

One basic thing that all quizzes, polls, and questionnaires often do wrong is that when they offer you responses, they also supply reasons for the response. For example:

Q. Do you pick up roadside litter?

A1. Yes, always; I can't stand that jerks are always leaving their trash around.
A2. Yes, sometimes, if I'm not in a hurry
A3. No, never--why should I pick up after other people?

All these answers have rationales that may not be the respondents' rationale at all. For instance, maybe you never pick up trash because you're immunocompromised and don't want to get sick. Or maybe you always pick up trash because you're interested in what things people discard. Or maybe you sometimes do, but it's not so much whether you're in a hurry or not but whether you have a free hand. And so on. (Never mind that the answers don't allow for certain meaningful other answers, such as "Yes, but only cans and bottles that I can collect the deposit on.")

If the rationale in an answer isn't your rationale, then you're forced to either choose that answer, even though it misrepresents your thinking, or not answer at all.

For that reason, I prefer questions that don't put rationales in the response choices .... But of course, the quiz creators would still be assuming the rationales, only their assumptions are hidden from you.

All of this is to say, always, always be skeptical of responses to questionnaires and polls. You don't know what kind of crap thinking went into their creation, how they were worded, and so on.

[personal profile] hyba 2020-11-12 02:53 pm (UTC)(link)
Great points! I remember that in uni they made a huge point of making sure we understood how to create surveys and undertake interviews so that when we designed them, our questions didn't lead respondents to a specific response or rationale. That would ultimately make at least some of your survey results inadmissible. For example, in the example you gave, nobody would pick the last choice because the rationale is phrased in such a way that it feels damaging to one's reputation - almost as though they aren't community-minded (which, depending on the society the respondent lives in, could be a huge no-no). So, naturally, the survey designer has led them away from that response.

So at the end of the day, what you had to do if you wanted to let everyone give them honest and detailed reasoning - and have more accurate results - is to create other questions that people can actually respond to in free-form. But that means more work afterwards, and you'd have to sift through all the responses and create categories for them to be able to analyze them statistically, and a lot of people are just too lazy to do that, I suppose.

[personal profile] hyba 2020-11-21 11:10 pm (UTC)(link)
I quite like how you put it, because it is true that today there's definitely a movement towards do-it-quick and minimize-costs that just... doesn't work when it comes to creating something that will last - whether that's research, a product, or an organization. It almost seems like a wasted effort, regardless of how minuscule. I'm starting to personally become increasingly inclined to slow down the world around me and take my time doing things well. Anything that requires some fast-paced churn-out at the expense of quality and really taking the time is probably not worth my effort.