Maybe the fact that we gravitated towards a career in market research makes us risk adverse? Market research is in fact putting in the due-diligence with data and feedback to ensure management is making decisions based on facts and evidence. The fact that a company would make large strategic and market decisions without doing a survey grinds our gears. It cuts us to the core. Don't go with your gut, go with the data! How can they take such a risk?
It's probably the same reason I dip my foot in the pool before jumping in, over-analyze my fantasy football team soaking in as much secondary research as I can find, and predict an NPS for all of my wife's dinners before I eat (secretly unless they appear to be promoters.) As a market researcher, we're inherently cautious, analytical, and careful. We want to ensure we've well-thought-out all possible assumptions and interpretations before moving ahead with a decision.
So it may come as no surprise that one of the most terrifying jobs as a research analyst is clicking send on an email invitations going to 1,000+, 10,000+, or 100,000+ respondents. What if the invite contains a glitch? What if the survey site crashes? What if there is a mistake? When we click our mouse on our laptops and click send, it comes with overwhelming anxiety but also great relief that the survey we've spent weeks or months perfecting is out the door.
On the flip side, perhaps the most exhilarating feeling from fielding a survey is when the first few survey completes start rolling in. What did they say? How did they rate this? I was right! I didn't expect that.
My reaction when an online survey invitation goes out
Some may deem a soft-launch or pre-test of surveys unnecessary. I've seen it. I've heard about it. It, in fact, does happen. Surveys go into the outbox everyday untested. Probably more so than companies are willing or want to say. Some claim it's not worth the additional day(s) or week(s) to refine the survey again. The time spent making revisions from a soft-launch is not worth the additional time it will now take to receive a final report. With any luck, this post will convince you to think otherwise or at the very least, make you think about dipping a toe into those cold survey waters before you jump head first next time.
What is a soft-launch?
Some pull a test sample by randomly selecting 1% or 2% of the total sample pool. Others select a batch of 100 random contacts. A soft-launch is a small pre-test or fielding of a survey to ensure the outcome of a full launches meets your objectives. Once the invites go out, there is no going back.
Here are 7 benefits to soft-launching an email survey.
Question comprehension
Surveys are a finely-tuned scientifically designed instrument. Really, they are. Everyone can write survey questions but those who have experience in survey writing know how to ask questions in certain ways, eliminate bias, order choices, and choose the right question design (grid, columns, radio buttons, etc.)
How you ask a question impacts how a respondent answers. It's no surprise respondents would be more likely to agree with this statement: "Pizza Hut meets its on-time delivery goal on 96% of its orders. Would you agree Pizza Hut cares about on-time delivery?" versus a more basic "Pizza Hut cares about on-time delivery." The additional language influences and biases responses.
Another example of poor question design is when your initial soft-launch returns questions with a high percentage of "other(s)" selected. If 45% of respondents are selecting other and typing in "social media", this tells you social media either needs to be added as a category or respondents are not understanding the question which asks "what types of non-digital marketing channels do you use in a typical week?"
Estimated response
This one seems straight-forward but it's a great way to estimate response rates to your survey. Although sample size is small you can anticipate open-rates (to see if the email language needs to be improved) and completion rates (how many partials versus full completes). So, if you are looking for 400 completes, you may not need to send invitations to your entire sample of customers, you may be able to pull a random smaller sample which will produce the 400 for you.
Estimated incidence or quotas
A large unknown in the online panel and online survey world is what percentage of a pool who will qualify for a study. In many cases the incidence rate of a customer satisfaction survey will approach 100% while the incidence rate of a highly targeted, highly specific B2B non-customer audience may dip as low as 10% or less. Before fully launching your survey, the soft-launch helps you understand how difficult or easy it will be to reach your targeted segment and fulfill quotas.
A/B Test
Perhaps the biggest impact on whether or not someone will take your survey through an email invitation is the subject line, followed by the initial first few sentences in the body. With a soft-launch it allows you to put your eggs in several baskets to determine what wording and language produces the best return. Without a soft-launch you'll be forced to put all of your eggs in one basket. Unfortunately, it may not be the best basket. This could cost you additional survey completes.
Misspellings or typos
Perhaps the most basic of the reasons to pre-test is this benefit. We live in a digital world that is go-go-go. Multiple priorities cross our inbox everyday and we are constantly being forced to multi-task. Survey designers make mistakes in the form of grammar errors, typos and misspellings. You and your team of 4 may not catch them but when you test your survey among 100 fresh eyes, a small typo will likely get noticed. The soft-launch provides you the opportunity to clean-up your survey before it goes to the masses.
Routing Issues
Depending on the complexity of your survey design, you may have advanced routing or logic built-into the programming. This programming could include several skip patterns. We've all been there. No matter how many times we test and test and test a survey, someone new comes in and finds a mistake in routing. It's better to catch this early than to sit down with your 400 completes you've received and realize that 275 skipped a question they shouldn't have.
Sample issues
List pulls and queries are not always 100% fool-proof either. We all make mistakes so if the IT or database analyst pulls a list thinking it was customers who have done business with a company in the past 12 months, it may be no surprise responses come back from the soft-launch saying they haven't done business in over 5 years. A market research study is only as good as the sample. Understanding the parameters of the sample and who you're surveying adds additional insight and interpretation to results. A soft-launch ensures the sample being uploaded for invitations is what was anticipated.
Drive Research is an online survey firm in Syracuse, NY. Our company offers a variety of Voice of Customer (VoC) and Customer Experience (CX) services to our clients. Interested in an online survey, customer satisfaction survey or community survey in Central New York? Contact us at 315-303-2040 or email us at [email protected].