BLOG

6 Types Of Marketing Testing Methods To Try

Published: 07/05/2020

There are many marketing testing methods out there, but which ones should you try? Here we look at 6 most popular ways to test various touchpoints across the entire user journey.

Marketing testing is probably one of the most misunderstood and thus, overlooked, areas in B2C marketing. Even when they are performed, marketing tests rarely influence any significant changes due to the fact that we either a) implemented the test wrong; or b) don’t know how to interpret the results that we’ve got.

We’ve all run an odd email subject line test at some point, or pushed our design team to A/B test different colors of a call-to-action button. But what really hides behind marketing tests and what mistakes should you avoid to enjoy data-driven insights? Join us as we dig deeper into the benefits of marketing and user journey testing.

What we’ll be covering:
1. What is testing in marketing?
2. Six popular marketing testing methods
2.1 A/B testing
2.2 Multivariate testing
2.3 User testing
2.4 Usability testing
2.5 Content testing
2.6 Incrementality testing
3. Why it pays to test the entire user journey

What is meant by testing in marketing?

​​Test marketing, also known as market testing, is a marketing technique that businesses use to evaluate the viability of their new products, services, or marketing campaigns before releasing them on a large scale. Businesses all around the world test multiple marketing scenarios to determine consumer needs and to find out whether the product, service or campaign meets and fulfils consumer requirements.

The test campaigns can be carried out in chosen test markets, such as in brick and mortar stores, online stores, or social media platforms. The experiment is generally done without customers’ knowledge to acquire non-biased feedback on the product or service.

After collecting test market feedback, the data is analyzed and data-backed insights are utilized to improve the product, service, or marketing strategy. To collect, aggregate and analyze the data, reporting tools are often used as these are able to automate data processes and showcase results in easy to understand visual reports.

Download the infographic | Yieldify

A/B and Multivariate testing

Let’s kick off with the tests that most of us are already familiar with: A/B tests and multivariate tests. These are user experience research methodologies used to compare variables and determine which set of variables is most effective. Also referred to as split testing, they are somewhat similar to one another but offer rather different outcomes.

What is A/B testing?

A/B testing compares two variations of the same element to determine which one produces better results. It is called A/B testing because it measures a user’s response to Variant A vs Variant B.

Marketing testing methods - AB testing | Yieldify
A/B testing = Same salad, different dressing

A/B tests have risen to stardom in the early 2000s (interesting tidbit – first cases of A/B testing date back to the 18th century and were used to treat scurvy). They were quickly adopted by designers, marketers, and developers alike to test UI elements, copywriting, discount offers, you name it!

The nice thing about A/B tests is that they’re fairly easy to implement and they can be used at a variety of touchpoints in the customer journey to test placement, color, copy, size, and other variables. But easy doesn’t mean they are always done right. Here are the main pitfalls of A/B testing:

  1. Testing more than two variants. The most common mistake is when you actually perform a multivariate test but judge the results as if it were an A/B test. To avoid this, remember this simple rule:

    [x] vs [x] = A/B test
    [y] vs [y] = A/B test
    [x] vs [y] – not an A/B test
    [x] vs [x+y] – not an A/B test
  1. Insufficient sample size. For an A/B test to return significant results, you must think of your sample size before running the test. This is not a one-size-fits-all situation – you really need to think of the level of confidence you want to achieve as well as the uplift rate you’re looking to get. There are dozens of sample size calculators online so find one that works best for you!
  2. Testing too short or too long. Again, it’s not easy to say what is a good test length for your A/B test but the general rule is that you should test for at least 7 days to reach statistical significance. If you haven’t, you can extend the test for another 7 days. However, if you run a test too long, there is a likelihood its results will compound with other tests and activities happening in the background.
Megabus A/B testing case study | Yieldify clients
See how megabus.com reached +7.5% CR uplift by A/B testing campaigns

What is multivariate testing?

Multivariate testing (MVT) compares multiple variations of several elements to determine which combination produces better results. The upside of multivariate tests is that you initially have more freedom and can make subtle changes rather than committing to versions A or B.

Multivariate testing = Different salads, different dressings

So what could be a multivariate testing example? Let’s say I want to increase conversions of my product page. Some of the elements I can use for a multivariate test are:

Or if you’re running an email marketing campaign to promote the release of a new product. You could test:

Note that MVT is a more advanced marketing testing method and is, therefore, most suitable for high traffic sites. Compared to A/B tests where traffic is usually split in two, in multivariate testing, it can be split in 3, 4, 5, and more groups.

User testing and usability testing

If you were ever involved in a product release, app launch, or even a website redesign project, you’ve probably heard of usability testing and user testing. Both of these terms refer to testing methods that involve real-life users and are often used interchangeably. However, they both offer different value and insights, and thus should be used separately.

What is user testing?

User testing, or user research, is an umbrella term that involves a variety of user testing methods: interviews, surveys, field studies, focus groups, and others. Its purpose is to determine the potential demand for an idea or a product by investigating user needs and behaviors. A typical outcome of a user test is a user persona or a target audience segment.

Under this definition, user testing would include usability testing.

User testing | Yieldify
User testing = Would you buy this brand new, cutting-edge salad?

What is usability testing?

Usability testing is a technique used to evaluate how easy or difficult it is to use a product or service by testing it on real users. Usability testing lets you understand how real users interact with the product or service, measure the user experience, and improve design based on your findings. In a typical usability test, participants are asked to complete a set of tasks while being observed by a research team.

Usability testing | Yieldify
Usability testing = How fast can you eat the salad with a fork vs. a knife?

“User Testing is about ‘Will this user use my product?’ Usability testing is to figure out ‘Can this user use this product?’ – the UX Blog.

There are multiple benefits of usability testing:

So what does this translate to for eCommerce marketers? Well, your websites and apps are your real estate, so focus on testing those! See how easy it is for users to find a product/offer they want, how they interact with ancillary products/offers on your store, how they navigate to the checkout, how they fill out the shipping form, etc. Usability testing can also work wonders for testing loyalty programs

Also – remember it doesn’t just have to be your product or service. You may as well test a competitor’s website to see where they might struggle or outshine you, gaining you a competitive advantage. 

There are several usability testing methods at play. Most commonly usability tests are split into: moderated vs. unmoderated; remote vs in-person; and explorative vs. assessment vs. comparative. The combinations go from there as you can have a moderated in-person assessment test, an unmoderated remote comparative test, and so on.

Usability testing methods | Yieldify
(via Hotjar)

Content testing

Arguably the most self-explanatory of all the marketing testing methods listed here, content testing can be considered part of usability testing. Its primary purpose is to determine whether a set of real-life users can understand and comprehend the content that you’ve provided.

Content testing | Yieldify
Content testing = Which menu is easier to read?

Again, the objective is to measure user experience: content testing is not to determine whether users like your content, it is to determine if they can read and understand it. A good example would be the user manual for assembling a piece of furniture: you can test if the text is legible, written accurately and whether it provides the information necessary to use the product. 

When thinking of eCommerce websites, content testing comes in handy when you need to determine the names of your product categories, phrase a discount offer and its terms and conditions or feature your headline or slogan on the page – does it catch attention and explain what the page is about?

Incrementality testing

One of the marketing testing methods that we like to talk about a lot at Yieldify is incrementality testing

Incrementality testing is a technique that measures the impact of a single variable on an individual user’s behavior. In other words, it helps to identify how effective a particular interaction is in influencing conversions. Incrementality tests are always performed with a control group to determine the uplift an interaction brings vs. a ‘no-change’ scenario.

Incrementality testing | Yieldify
Incrementality test process explained

Incrementality testing falls under the A/B testing umbrella. However, whereas A/B testing is used mostly on creative elements and measured by click-through rate (CTR), incrementality testing focused on sales uplift and is measured by conversion rate (CVR).

Widely used in digital advertising, incrementality testing found its way into conversion rate and customer journey optimization processes as well. At Yieldify, we run incrementality tests to help our clients understand the value of the campaigns that we run on their websites. Here’s an example of what an incrementality test report could look like:

Incrementality report example | Yieldify
Incrementality test report example

Why it pays to test the entire user journey

The marketing testing methods we’ve discussed above are incredible ways to finetune your product, software or service. By employing some of them together you’re bound to collect invaluable data about your users and make necessary changes to boost your conversions.

Nevertheless, conversion rate optimization (CRO) is just one side of the coin. If you really want to see significant outcomes of user testing, you have to employ these practices across multiple user touchpoints. In other words, you must test the entire user journey, from initial marketing channels, all the way through to how potential customers navigate your website.

The main difference between CRO and customer journey optimization (or CXO – customer experience optimization) is that CRO tests individual element sets removed from the rest of the customer journey. Whereas CJO (or CXO) runs across multiple touchpoints providing a ‘big-picture’ look at the different levers you can pull to improve your customer journey as a whole.

For that purpose, marketers exercise customer journey mapping. This allows them to not only develop empathy for the customer, but also understand consumer behavior patterns that influence conversions, loyalty, and a number of other business metrics that are directly connected to revenue.

In conclusion

Regardless of the type of market test you are carrying out, the most important thing is that you are conducting tests, collecting copious market research, and thus, gaining access to all of the valuable insights and data that can better inform your strategy. 

For the best results, it is clear that the holistic, 360 degree, all-encompassing, customer journey map reigns king. With high-impact benefits such as higher conversions, loyalty and revenue income, the marketers that can include this potent weapon in their arsenal will ultimately reap the best rewards and edge ever further in front of the competition by better understanding their target market.

Marketing Testing FAQs

What is a marketing testing method?

Marketing testing methods include A/B testing, Multivariate testing, user, and usability testing. Each testing method is designed to improve performance and user experience.

What is meant by testing in marketing?

Testing in marketing can mean a lot of different things depending on the type of tests you are running. It could be a limited release of a product to a test market, or an A/B test showing two different product pages to a select group of users.

Click to download

Marketing and user journey testing infographic | Yieldify