How to Use a Survey to Conduct A/B Testing for Ads

ipad with a/b test survey on it

As a marketer, you’re likely no stranger to A/B testing an advertisement. 

For those who need a reminder, it is when a brand launches two advertisements at once. Each ad has a component that is slightly different from the other, such as a different color scheme or different ad copy.

After a day or two, the brand looks at key analytics to determine which ad is offering the better return. 

This is the traditional, more common way of A/B testing in marketing. However, there is a newer, more data-driven approach to achieving the same goal: using a survey to conduct an A/B image test.

In this blog post, our market research company explains a recent project completed where we conducted an online survey to determine what advertisement was most appealing to a targeted demographic.

Read: Ultimate Guide to Ad Concept Testing Surveys

What is A/B Concept Testing?

A/B testing, also a form of qualitative research, is a procedure in which 2 or more variables or versions are tested against each other in order to determine which concept would perform the best.

Normally, this is done through a process in which a target audience is identified and sent a survey – usually online or through email.

Respondents are asked a series of questions for each concept to be tested. The questions are generally geared toward thoughts, attitudes, feelings, and behaviors towards each version presented.

Examples questions could include:

  • Which version captures your attention the most, A or B?
  • On a scale of 1 to 10, how visually appealing do you find each version?
  • What is the first thing that comes to mind when you see version A?

At the conclusion of the survey, the results are compiled for each concept that is tested, and the one with the highest overall score (or strongest feedback) would be recommended for deployment.

Benefits of A/B Testing with Surveys

There are several benefits to using A/B testing including:

  • Using findings to increase user awareness, conversion rates, or engagement with your brand
  • Minimizing risk to better manage design costs by testing concepts vs deploying several concepts without properly testing
  • Reducing bounce rates in situations where a website or online content is tested
  • The ability to change variables such as ad creative, audiences, or placement to determine which concept or strategy performs best currently, and for future campaign usage

For more benefits of concept testing with surveys, watch our video below.

Example of a A/B Image Test Survey

A healthcare manufacturer is launching a campaign for a new product. The client wanted to create an advertising campaign that would resonate most with their target audience.

The main goals of the advertisement included:

  • Position the product as a leader in the space
  • Drive awareness of the new product
  • Learn how to increase interest in a trial of the product. 

However, they were unsure what approach was best to take. Instead of spending money on two advertising concepts, the healthcare manufacturer wanted to conduct a survey to first identify a winning concept. 

The results would allow them to focus on completely designing and executing one ad. This was a better alternative to wasting time and money on Ad Concept A and Ad Concept B, only to have Ad Concept B run for a few days. 

A/B Testing Survey Approach

To address the objectives at hand, our market research firm recommended using an online survey to A/B test two graphic ads. 

The survey included:

  • Image selections with optional follow-up comments for positive and negative context and sentiment
  • Open-ended questions for themes and drivers of choice.
  • Heat maps to showcase pieces of the most appealing ads.

The survey targeted and used screening questions to ensure the intended audience was reached.

Recommended Reading: 5 Steps to Conducting an Ad Concept Testing Survey

A/B Testing Survey Questions

Below is an overview of example ad concept testing survey questions. These questions include placeholders to keep the topic and client confidential. 

  • Which of the following best describes the industry in which your organization operates? Select one. 
  • Which of the following best describes your position/role? Select one.
  • Which of the following specialties applies to you? Select one.
  • How many years have you been a [insert role]? Select one.
  • Do you have experience with [insert task]? Select one.
  • In a typical month, approximately how many [insert task] do you complete? Enter your response below.
  • How much influence do you have on the purchasing of [insert product]? Select a rating.
  • Which of the following [insert product] are you aware of? Select all that apply. 
  • If purchasing a [insert product], how important are each of the following factors? Select one for each.
  • What would convince you to try another manufacturer of [insert product]? Enter your response below.
  • Please review the advertisement below and share what you like or dislike most about it. Click on a part of the image to provide feedback on that area. Then select ‘Like’ or ‘Dislike,’ enter any comments, and hit ‘Save’ when done. Select another point on the image to share additional feedback. 
  • How appealing is the advertisement below to you? Select a rating.
  • Which of the following best describes the product shown in the advertisement below? Each row represents two opposing ideas. Select a point closest to the idea that best describes the advertisement.
  • Which of the two advertisements below do you prefer? Select one.
  • Thinking about the previous advertisements, how appealing are the following messages regarding the [insert product]? Rank on a scale of 1 to 6, where ‘1’ is ‘most appealing’ and ‘6’ is ‘least appealing.’

A/B Test Survey Results

While the results remain confidential with the client, the chosen reporting options included a topline summary report.

This type of report includes 3 to 4 pages of key findings in a bullet point format, online reports which feature the question-by-question results, and export of the raw data file. 

The topline summary report also included sections for the background and objectives, approach, online reports, demographics, awareness and usage, ad recommendations, review of ad 1, and a review of ad 2. 

Details of the A/B image test survey:

  • The survey took an average of 10 minutes to complete and included 37 questions.
  • The survey received 75 responses.
  • Fieldwork for the survey began on July 10 and lasted until July 20, 2021. 
  • With a probabilistic sample, a total of 75 responses at the 95% confidence level offers an 11% margin of error.
  • If the survey were conducted with another random pool of 75 respondents, the results would yield within +11% or -11% of the stated totals in the reports.

Conduct an A/B Test Survey with Drive Research

Has your business thought about using a new logo or designing a new ad creative in order to enhance your image? If so, A/B concept testing could be for you!

And we can help.

Drive Research is a national market research company specializing in the advertising and marketing sector. Our team of certified professionals partners with brands across the country to execute ad concept testing research such as A/B testing surveys.

Interested in learning more about our services? Contact our team today.

  1. Message us on our website
  2. Email us at [email protected]
  3. Call us at 888-725-DATA
  4. Text us at 315-303-2040

emily taylor about the author

Emily Taylor

As a Senior Research Analyst, Emily is approaching a decade of experience in the market research industry and loves to challenge the status quo. She is a certified VoC professional with a passion for storytelling.

Learn more about Emily, here.

subscribe to our blog


Online Surveys