With the emergence of online and mobile surveys particularly through market research panels, online data quality has become a chief concern for market researchers. Respondents are time-poor and should they choose to participate in surveys, they want to the research to be easy and fast. However, there is such a thing as surveys being completed too fast.
What is survey satisficing?
Survey satisficing is when the survey respondent takes short-cuts to complete a survey faster, knowingly clicking incorrect answers in order to finish quickly. Market research analysts are trained to prepare for these survey satisficers and develop standards into each survey to counter-act these data fiends. Data quality checklists can help identify these survey respondents and remove them from your data file before analysis.
What is an example of survey satisficing?
The online survey participant Tom is asked: "Have you been to McDonald's in the past 3 months?" - with an option to click yes or no. Tom had been there last night and immediately clicks "Yes". Tom is then asked a series of questions about satisfaction, purchases, perception of the restaurant, likelihood to go again, and a laundry list of other questions regarding McDonald's that takes him 5 minutes to complete. Once finished with that series he is asked: "Have you been to Burger King in the past 3 months?" Even though Tom went last week, he opts to click "No" to avoid the additional 5 minute series of follow-up questions to complete the survey faster and earn his 40 panel reward points.
A study conducted by Jon Krosnick and David Vannette at Stanford University in 2013 identified two key drivers of satisficing in surveys: (1) lack of motivation from respondent and (2) the difficulty of the task. Survey satisficing occurred the most when the respondent had little motivation and the task was deemed difficult.
What measures can be put in place to prevent survey satisficing?
Here are 5 tips to prevent respondents like Tom from ruining your data quality:
Option 1: Drop a Red Herring in Your Survey
This can come in the form of a listing a brand which does not exist when asking awareness questions or even something as blunt as asking the respondent to click answer "C" in a series of grid questions. Adding this in your online survey helps identify speedsters.
Option 2: Using Extreme Opposites
This is one we don't usually like at Drive because it can involve double negatives. However, many survey writers and designers use it. This would involve asking a respondent to rate the level of agreement from strongly agree to strongly disagree on the following: "The Big Mac is the most tasty menu item at McDonald's." To monitor data quality the survey would then ask the opposite later in the series: "The Big Mac is the least tasty menu item at McDonald's." If the respondent rated both strongly agree or both strongly disagree it will raise a red flag in your analysis because both options cannot be true.
Option 3: Check for straightlining
Survey takers hate long series of 1 to 10 rating scales. Repetition is the killer of survey engagement. Once the survey participant reaches his or her threshold, they may begin clicking just clicking "9". Keep an eye out for these straightliners and remove them from your data set.
Option 4: Review Time to Complete the Survey
You've most likely tested your survey thoroughly and as the survey designer, you know what range of time it will take someone to complete it. During fieldwork you can even calculate your average time to complete. If the survey takes an average of 12 minutes and you have a handful of respondents that have completed it in 3 to 5 minutes, take a closer look.
Option 5: Limit the Number of Selections
In an online survey it is important to be exhaustive and list all the possible answer choices but don't let this get out of hand. This is especially important when designing your survey for mobile devices. When listing the number of fast food restaurants respondents might be aware of, you are better off selecting the top 6 to 8 locations and listing an "other(s)" to save time. If you have a list of 78 fast food restaurants which the participant must select from, you will not only invite satisficing but you will also invite dropout.
Drive Research is an online survey company located in Upstate, NY. We serve the market research needs of clients in Buffalo, Rochester, and Albany. Our market research services include online surveys, phone surveys, and other options. Contact Drive Research at firstname.lastname@example.org or call us at 315-303-2040.