Data quality is always a hot topic when it comes to online surveys. It can feel defeating when a researcher meticulously designs a survey only to find poor data quality lurking in the responses.
There are 4 types of online survey respondents that scare the $&#@ out of researchers.
- The speeder
- The professional
- The one who doesn’t read directions
- The tech wiz
Keep reading to learn how to spot poor quality survey responses, what causes illegitimate data, and five unique survey questions to ask that will clearly identify spammy respondents.
How to Spot Poor Quality Survey Respondents
Ensuring high data quality is critical for online research. There are a few ways researchers can stand out against fake survey fools to ensure results are accurate and trustworthy.
A few ways researchers spot poor-quality survey respondents include:
- Asking quality-check questions throughout the survey
- Ensuring all respondents are unique
- Reviewing how long the survey took respondents to complete
- Ensuring respondents are not straightlining
- Reviewing all open-ended questions
Watch our video for more survey data cleaning tips.
💡 The Key Takeaway: Reviewing data quality is critical before analyzing survey results. The five factors described above provide an overview of how market researchers review survey data quality.
Recommended Reading: 4 Basic Tips to Quality Check a Data File in Market Research
What Causes Bad Survey Respondents
Safe to say there are many reasons for poor data quality.
Common reasons stem from...
- Invalid survey traffic
- Poor survey design
- Skipping data quality actions to prevent illegitimate respondents
While reviewing data quality may take a few hours to complete, it’s preferable to take the time needed to ensure high data quality rather than end up with nonsensical data mixed in.
1. Invalid survey traffic
This refers to bot or traffic that should be routed out of your survey. This can happen for various reasons, but it’s important to determine if this affects your survey.
A few ways to prevent this issue is to ask a reCAPTCHA question at the beginning of a survey to ensure no bots enter and include a duplicate cookie and/or IP address prevention which ensures one response per computer.
Some platforms also allow users to make the survey link not indexable on search engines, which also can decrease invalid/bot traffic from finding your survey.
2. Poor survey design
When assessing data quality, ensure that respondents will not be able to contradict themselves within the survey unintentionally.
Set your survey up for success by ensuring your survey is properly designed and ensuring respondents won’t be able to enter contradictory data.
3. Skipping data quality questions
It is best practice to include at least a few data quality questions in a survey, particularly if it is going to a random sample. These are also known as red herring questions.
Don't worry, we have some great red herring question examples below.
💡 The Key Takeaway: Protect yourself from poor quality survey data by taking measures to inhibit invalid survey traffic, using best practices for survey design, and including data quality questions throughout the survey.
Survey Questions that Identify Poor Responses
Below our online survey company shares unique questions to include in your questionnaire that will easily spot poor survey respondents.
1. Include a reCAPTCHA
Many survey platforms allow users to add a reCAPTCHA question to their survey. We typically add a quick note before the reCAPTCHA saying, “First things first! We need to know you're not a robot.”
2. Ask a comprehension question
Example 1: Which of the following words best relates to the word shirt? Select one. [Programmer note: Randomize]
- Truck [Disqualify]
- Pants
- Apple [Disqualify]
- Sidewalk [Disqualify]
Example 2: Match the words below with the corresponding category. [Programmer note: Disqualify if fruit does not exactly match “Apple” and “Pear” and house does not exactly match “Garage” and “Yard”]
Category 1: Fruit
Category 2: House
- Apple
- Pear
- Garage
- Yard
3. Ask them to do something specific
Example 1: Enter the number between 21 and 23. Enter your response below. [Programmer note: Disqualify if does not exactly match “22”]
Example 2: Select the colors of the American flag. Select all that apply. [Programmer note: Randomize]
- Red
- Orange
- Yellow
- Green
- Blue
- Purple
- Pink
- White
- Black
4. Include a red herring (or a couple!)
This can mean asking respondents to select a specific answer, or including a fake response choice that disqualifies the respondent if selected. Below are a few red herring question examples.
Example 1: Select the ice cream flavor.
- Rocky road
- Penguins
- House
- Car
Example 1: What is your favorite color?
- Pink
- Blue
- Yellow
- Soccer
5. Ask an articulation question
If it makes sense in your survey, consider asking an open-ended question that helps ensure respondents are knowledgeable about the survey topic.
It’s important to note that open-ended questions add to respondent fatigue, so this may not be a great fit for every survey or you may want to make the question skippable.
More Ways to Spot Poor Survey Respondents
1. Remove duplicate IP addresses
Many survey platforms allow users to export users’ IP address, which can be reviewed to ensure each IP address is unique (i.e., each respondent came from a unique computer).
2. Review time to complete
A general rule of thumb to find speeders is to delete those who complete a survey in less than 25% of the time it took to complete the survey on average.
For example, if the survey took 8 minutes to complete on average, this would mean deleting anyone who completed the survey in 2 minutes or less.
The big caveat is that it’s important to ensure you are not artificially changing your results by favoring surveys that include more questions due to routing. Before deleting speeders, ensure those you remove did not take a shorter survey path.
3. Check for straightliners
Straightlining is someone who enters the same response through a grid question or question list.
Including two red herrings in a grid question can help ensure straightliners are removed from your survey.
4. Review all open-ended responses
While this one typically takes the longest to execute, it’s important to ensure all open-ended text from respondents is reviewed (consider all please specify or open-ended text box).
This helps ensure no responses need to be recoded or deleted.
Learn more in our blog post, Approaches to Analyzing Open-Ended Survey Responses.
💡 The Key Takeaway: While reviewing data quality can be time-consuming, it’s important to include this step in the research process before data is analyzed.
Contact Our Online Survey Company
Drive Research is a full-service market research company specializing in online surveys and data cleaning best practices. We understand the value of using data to drive business decisions, but it's even more important that the insights used are accurate and high-quality.
Learn more about our market research services by contacting our team today.
- Message us on our website
- Email us at [email protected]
- Call us at 888-725-DATA
- Text us at 315-303-2040
Emily Taylor
As a Senior Research Analyst, Emily is approaching a decade of experience in the market research industry and loves to challenge the status quo. She is a certified VoC professional with a passion for storytelling.
Learn more about Emily, here.
