By this point, I'm sure you've seen the quote circulating social media about using an amateur and not a professional. It's something along the lines of "if you think using a professional is too expensive, wait until you hire an amateur."
It shows 2 pictures side-by-side of a tattoo artist inking a tattoo on to someone's back. One is a picture of a unicorn that looks like it is drawn by a student in Ms. Miller's 1st grade class. The other is much more extravagant and detailed by the true artist. Much like market research, the buyer won't understand the difference between "amateur" and "professional", "high quality" and "low quality", "good" and "bad" until after the project has been completed. At that point they understand the value and ROI between an amateur and professional.
It Takes 10,000 Hours to Master a Skill
Like any professional service, to master it, it takes years of experience, training, and hands-on work. In the book Outliers, the author Malcolm Gladwell makes the claim it takes 10,000 hours to master a skill. Assuming most work an average of 8 hours per day, that 1,250 days straight of working on one skill. That's nearly 3 and a half years straight of perfecting a specific skill. That's a long time.
In the world of market research, in order to master survey writing that's 3 and a half years of work. So it should come as no surprise that there is a huge difference when you work with a market research company versus trying to complete a project in-house.
If you completed the survey in-house, could you construct a decent survey which will provide you answers to some of your organization's questions? Sometimes, yes. But there are likely several factors of survey design and construction you have not considered, planned for, or addressed in your survey script.
We cover those in this post, in the science of survey writing.
We'll teach you about the science of survey writing.
What do to. What not to do. It's a science. No lab coat required.
Factor 1: Sequence of Questions
Order and flow matters. Our market research company in Buffalo always recommends creating an outline first before you even write question 1 of your survey. The sequence and flow of a survey matters and you'll want to tackle this first. The outline looks and the big picture of laying out a survey before you worry about the more tactical factors within each question.
Understand that asking questions in a specific order will have an impact on results. For instance, if you want to ask about customer satisfaction on a 1 to 5 scale, should you ask it early in your survey or late in your survey?
If you ask it early, you will get a response top-of-mind before you dive into more specific questions about the customer experience. If you ask about it later in the survey, you've already biased what the customer experience entails by asking about factors that are important in choice and even feedback on specific topics (e.g. price, customer service, responsiveness, etc.)
Our market research company in Buffalo favors asking questions like customer satisfaction and Net Promoter Score (NPS) early in the survey. By adopting this mentality the loyalty feedback is more top-of-mind without having the score influenced by other factors tested in the market research.
Asking this early means "loyalty" is self-defined by the respondent. However, if you ask it later in the survey, you will define what encompasses "loyalty" by the other questions you ask.
This is the type of flow and sequence of questions only one with experience in the field understands. Working with a professional and a market research company will help your organization outline the best possible survey to address your needs.
Factor 2: Eliminating Bias
Bias. A word in market research that sends shivers down a Research Analyst's spine. Rightfully so too. Bias is the enemy in market research. It's the kryptonite to our Superman. It's the Joker to our Batman. It's the New York Giants to the New England Patriots. Sorry Patriots fans, I had to throw that dig in there. #2WinsForEli
Bias is a chameleon. It comes in many forms from sequence bias, order bias, response bias, methodology bias, and so many more. It's the job of a market research company to extract and eliminate potential bias in the survey scripts they design.
After a while and one begins to master the science of survey writing, this becomes second nature to professionals. Order bias is addressed in the next factor. Response bias can be mitigated by utilizing mixed-mode approaches. If you only send a customer satisfaction out to those who have an email you will likely bias results to those who are more tech-savvy. Finding ways to survey those without email addresses (phone or mail) helps eliminate this.
This is one of the factors that goes unnoticed with amateur survey writers. It can have a large negative impact on your results and send your team in a wrong direction. Bias lurks its ugly head throughout the market research process, but never more than during survey design.
Factor 3: Ordering Responses
The ordering of responses plays a large role in writing a good survey but also helps to eliminate bias. By eliminating bias it makes your results more reliable and accurate. When it comes to ordering responses, you need to think about several options: inverse ordering and randomization to name a few.
Inverse ordering is the process of rotating the order of responses for each respondent. One respondent would see the answers as A, B, C, D and the next would see it as D, C, B, A. This inverse order works well when you list answer choices that follow a logical sequence: highly satisfied to highly dissatisfied. Another would be time series: more than once a month to less often than once every 3 months. In those situations where scaling and sequence are important, it would not make sense to randomize those options.
Randomization is the process of rotating answer categories in a non-logical pattern. One respondent would see the choices as B, D, A, C and the next would see this as D, A, C, B. This is used where the order of choices does not make a difference such as source(s) of marketing awareness: Facebook, Television, Billboards, Websites, etc.
Writing a market research survey? Think about getting some advice from a market research company. They'll consider these 5 factors and more to ensure your survey obtains accurate and reliable results to guide your decision-making.
Factor 4: Balancing Open and Closed-Ended Questions
Engagement is key. If your survey contains too many questions or takes too much time for the respondent, you will likely see a high number of drop-outs or poor quality data. With that being said, our market research company is always cognizant of the number of open-ends in our scripts.
Open-ended questions require the respondent to type out a response. This takes additional time and thought to answer. If you can list answer categories for respondents to choose from, it makes the survey experience that much easier. However, in some cases you may not want to bias results or answer choices.
An example would be "What is the first word or phrase that comes to mind when you think of Company ABC?" Listing all of the possible answer categories here would be impossible. If you did list 5 to 10 choices it would put extreme bias on the data and likely result in respondents selecting 1 of the 10 choices they may not have thought of or listed if it was open-ended.
As best practice we try to not ask more than 2 to 3 open-ended questions per 15 to 20 questions in a survey. Limiting the number of open-ends provides very high response and completion rates on our market research company's surveys. Don't overdo it when it comes to asking open-ended questions or you'll pay for it with lots of skips and poor quality data.
Factor 5: MECE
The first lesson in market research training. MECE stands for Mutually Exclusive and Collectively Exhaustive. Mutually exclusive means all of the responses listed in your closed-ended responses are separate from one another with no overlap. Which of the following do you select if you are 35?
Right: Under 25, 25 to 34, 35 to 44
Wrong: 25 and under, 25 to 35, 35 to 44
Collectively exhaustive stands for ensuring that all of the answer categories you have listed exhaust the options of all potential responses. For example, let's say you ask your survey participants where they have seen or heard about your brand in the past 3 months?
Right: Billboards, Television, Radio, Social Media, Other(s)
Wrong: Billboards, Television, Radio
Being collectively exhaustive means you provide respondents with the realm of all possible answers. Adding an "Other(s)" ensures you cover any additional categories you have have missed. When we say "all possible" we mean listing 8 to 10 of the major categories with an other. We do not recommend extending a list beyond 8 to 10 categories because it will produce a lot of drop-out on your survey. Choose the 8 to 10 most popular choices.
Contact Our Market Research Company
As a market research company in Buffalo and Upstate New York, we know a few things about writing surveys. Interested in learning more about market research best practices and survey writing? Subscribe to our blog in the right toolbar by giving us your email. We'll send you a new blog post every time we publish.
Questions about an upcoming market research survey? Contact us at [email protected]. You can also use this contact form or call us at 315-303-2040.