How to Be Objective and Avoid Survey Bias

[…]

5 Examples of Survey Bias and How to Address Them

Wouldn’t you agree that most survey questions are biased? 

What was the first thing that popped into your mind? Was it something like, “Well, I hadn’t really thought about it until you asked, but I guess so?” Then, you fell victim to a form of survey bias. The question pushes people toward the answer that you believe is true.

Survey bias can skew your results and make your research unreliable. In this post, we’ll explain survey bias and review common biased questions and how to avoid them. 

What is a Biased Survey?

Biased surveys contain poorly constructed questions that influence results. These questions either steer respondents toward a certain answer, are too vague or confusing. 

• Biased: How amazing was your experience with our sales team?
• Broad: How does our food compare to competitors?

Introducing bias into your survey through poorly worded questions can make your results unreliable. 

Recommended: Ranking Scale vs. Rating Scale – Which Question Type Is Best?

Why You Should Avoid Survey Bias

Most companies don’t conduct research just for kicks. Typically, it guides the decision-making process, like helping a marketing agency decide which ad creative to use for a client campaign.

In our article on concept testing, we talked about how companies utilize agile research to ensure an idea or product is worth investing time and money in. During the concept testing phase, companies need unbiased and accurate feedback from consumers. By including biased questions, results can be skewed and cause a company to develop a new product that will ultimately fail. 

Another consequence of biased questions is reduced response rates. Today’s consumers are pretty savvy. If your questions are confusing or clearly leaning one way, people are more likely to abandon the survey before completing it. 

5 Examples of Biased Questions and How to Fix Them

Now that we’ve covered why biased questions are bad, let’s look at how to avoid them. Below are the five most common examples of biased survey questions. 

Leading Questions

Leading questions plant a seed in the respondent’s mind that shapes their answer. The way you ask the question influences the person’s opinion, and you no longer get their genuine reaction.

Leading Question Examples	How to Fix It
How much will gas prices increase in the new year?	The question assumes it’s a matter of fact that gas prices will increase. Better ways to ask this question include: – Do you think gas prices will increase in the new year? – How do you think gas prices will change next year?
How great was your experience at our restaurant?	The phrasing implies people should have a great experience, making people who had a less than stellar experience feel awkward about answering honestly. Try these options instead: – Describe your experience at our restaurant.  – How would you rate your experience at our restaurant?

 

Double Barreled Questions

This type of question asks respondents about two things, but only allows one response. The question assumes people have the same opinion on both items, but that’s not always the case. 

Double Barreled Question Example	How to Fix It
How satisfied were you with your food and service?	People can have a lousy meal and an excellent server or vice versa. If you want to evaluate opinions of both the food and service at a restaurant, split the question into two. – How would you rate your food? – How would you rate your service?

 

Double Negative Question

No means no, except when it means yes. Confused? So are we. If you’re unsure what’s being asked, respondents definitely will be too. With these questions, the double-negative in the question makes it confusing if respondents should pick yes or no. 

Double Negative Question Example	How to Fix It
I don’t think people should not buy new instead of used cars. Yes or No?	Most people would have to read this statement a few times to figure out how to answer it. Better ways to ask the question without a double negative include: – Do you think people should buy new or used cars? – I think people should buy used cars instead of new cars. Strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree, strongly disagree.

 

Loaded Questions

These questions make assumptions about respondents that may or may not be true. This skews your results by forcing respondents to ultimately accept or reject an assumption. 

Loaded Question Example	How to Fix It
What’s your favorite cocktail?	This question assumes people drink. Even if the person responds with I don’t have one, you still won’t know if it’s because they don’t have a preference or don’t drink at all. Use branched survey questions to identify if someone drinks alcohol before asking the question.  – Do you drink cocktails? – Only for the people who answer yes, ask, What’s your favorite cocktail?

 

Absolute Questions

These questions force respondents to agree with absolute statements like always and never. The problem is there are few absolutes in life. For example, respondents struggle saying they never do something if they’ve done it a couple of times. 

Absolute Question Example	How to Fix It
Do you always brush your teeth twice a day?	This question is problematic on two fronts. First, even people with the best dental hygiene might forget to brush their teeth every once and a while. Second, people might be embarrassed to answer no. Instead, try these questions: – How often do you brush your teeth on an average day? – How many days a week do you brush your teeth two or more times?

 

Getting Help Building Better Surveys

Use these examples when reviewing your next survey to avoid asking biased questions. In general, to get the most accurate results and provide the best experience for your respondents, a survey should:

• Use objective questions. 
• Provide clear and easy-to-understand choices. 
• Be brief and concise. 

If you want help building better surveys, download our ebook on DIY surveys for more tips on creating rock-solid surveys.