Become an email & SMS marketing pro

Subscribe to get news, tips and updates delivered to your inbox.

Notify me on Messenger
8 min.

We explored the A/B test and the fundamental KPIs (open, click, and response rates) to understand what aspects to experiment with between two variants of the same email. Objective: increase ROI of campaigns.

The A/B test is an exercise in humility for even the most experienced marketer. On one hand, testing means researching what works best, and on the other, questioning what has been done up to now. Questioning means knowing how to change and evolve, which is the very principle of marketing.

In this post we want to explain the theory, but even more so the practice of this fundamental email marketing activity.

What is an A/B test?

An A/B test, also known as split testing, means submitting two or more different versions of the same message to a sample of recipients, analyzing the reaction to each version, and determining which one is most effective. The goal is to optimize individual aspects of the email with the aim of improving performance not just by following intuition or common sense, but based on actual audience response.

In God we trust. All others must bring data.
William Edwards Deming

In a two-variant A/B test:

  • The “neutral” version is called the control
  • The variant containing the changed element to be tested is called the treatment

Objectives of the A/B Test

Now let’s take a look at the A/B test from the perspective that is most important to marketers. We have identified four fundamental KPIs, and we have turned them inside out to analyze the elements which would allow an A/B test to improve email performance.

Improve open rates

The performance of an email’s subject is influenced by subtle nuances, which must be carefully weighed. The elements on which to perform the tests are as follows:


Changing one or two words is enough to give a decisive boost to the open rate. Here is an example using an imaginary app as the sender for a job search:

A: Test Jobs available with a 50% discount
B: Test Jobs available to establish a career


Open rate

Subject B had a +13.3% open rate compared to subject A

Customers are more excited about the idea of ‚Äč‚Äč”establishing a career” than being enticed by a discount.

Approach: affirmative or interrogative?

As a first step to the winner’s podium, the subjects of the emails sent by former US President Barack Obama certainly took note of this:

However, what if the White House had opted instead for the interrogative versionHey?¬†– would the campaign have turned out the same way? We will never know, but all marketers know the effectiveness of an interrogative subject line. Creating A/B tests on this text is essential: for example, try comparing “Create your free profile today!” with “Do you want to create a free profile?”


Many marketers recommend keeping the subject line between 28 and 38 characters; however, this should not be taken as an excuse not to test, as there are so many variables. We know that iPhones only show 35-38 characters in portrait mode (80 horizontally), and the number varies (often decreasing) based on the size of different screens. On desktops, on the other hand, it is possible to provide a more detailed description and be more convincing. That is why it is always important to perform A/ B tests to compare a more concise and a more conversational version.


From length, we can now move on to the degree of customization: try conducting an A/B test comparing a generic, common subject, with one that has a dynamic field that automatically enters the recipient’s name. You can also speak directly to the user, on a first name basis.

Find out how to customize emails with dynamic fields >>

The second level of the subject: the preheader

The preheader is that short text that is displayed after the subject in the recipient’s inbox which gives the recipient an extra chance to get an idea of ‚Äč‚Äčthe email’s content.

The preheader is not just a service element available to recipients, but a strategic element that works to convince users to open the email. Including and improving a preheader leads to an increase in open rates, CTRs, and conversions. For this reason, it is advisable to also experiment with this level of content.

Learn all about email preheaders >>

The effectiveness of emojis

Small but showy, frivolous but attractive. Emojis are ambiguous elements that are increasingly used in brand communication.

In this realm as well, the advice is to test if open rates show an upsurge with emoji use. You can also compare different emojis to understand which one is most effective. One thing is certain: some are used more often, as evidenced by this study of the most used emojis in email subjects:

Source: MailChimp

Improving CTR

We are now at the deepest level of the funnel: the click rate, which is most likely the most important metric to marketers. Here most of the text will inevitably hinge on the call-to-action, that focal point of the email that can be likened to the “waiting room” of conversion.


Using contrasting colors, in reference to the email color scheme, is one solution for distinguishing the action to be performed. As a word of caution, don’t overdo it: one wrong choice could ruin the whole communication, and push clicks away.

Also consider that it is always a good idea to reference the corporate brand identity colors within an email.


The size of the button can make a difference when it comes to CTA. The button should be big enough to be obvious, but if excessively large, it could be deemed too aggressive, thereby hindering the user experience.

The display on mobile devices is also an element to consider when choosing a button size. Even in this case, testing is best for understanding what is most effective.


One example is the classicBuy now“. Clear and concise, this CTA could, however, worry less accustomed readers by suggesting a “forced” purchase just after the click. In this case we recommend testing softer solutions, such as “Add to Cart” or “Continue with purchase”. This is just an example to help you understand how text is also important for driving clicks and conversions.

Special effects

Buttons can be highlighted with particular shapes. The variables could involve:

  • The roundness or angularity of the corners
  • Three-dimensional effects or a flat design
  • Shadows or lack thereof

The number of CTAs

There are campaigns where goals lead to a dilemma, in the sense that they diverge and force the marketer to make a choice: focus on a single call to action or dare to add two? The A/B test will give you a more or less unequivocal verdict.

Improve sending times

Identifying the right time for email delivery works as a driving force for open and click rates. Well-established hours in the middle of the day are not necessarily the best. Depending on the type of business, the sector of activity, and the target audience, the best times can vary greatly: in general, B2B companies get better performance during “desk” business hours, while B2Cs get better performance at all other times.

The best thing is to perform an A/B test for each of your sending categories (newsletter or DEM) to find out:

  • The best day of the week
  • The best time slot

Improving response rate

This is more of a niche metric, as email marketing campaigns that target the recipient’s response are not as common. For many, however, such as sales people, this is a fundamental indicator. Here are some points on which to carry out the all important A/B tests.

Comprehensive or concise body copy

Can your emails be scanned and understood in 8 seconds? If so, great, otherwise you risk losing the attention of your recipients.

Take an email template you have sent in the past and cut its content. You can do it yourself or let an application such as Hemingway help you. Then take the two versions of the email and compare them with an A/B test.

Degree of customization

As already mentioned above for the subject, customization with dynamic fields is now a must in email marketing. Try testing this element in the body of the email as well, calling the recipient by their name, including their company name, or customizing more in depth by specifying the recipient’s business role.

Conversational formulas

This is an even larger, more complex field, and is even more difficult to measure.

Many marketers and sales consultants use templates with predefined text. Often a few small tweaks or the reorganization of content are all that is needed to get increased response rates.

To sum up

Testing is part of the DNA of email marketing. It is a highly immediate tool which only requires a few steps to compare different versions of a campaign.

If you haven’t tried MailUp yet, we suggest requesting a free 30-day trial of the platform. In just a few clicks you can experience what works best for your campaign and improve performance.

Give MailUp a go!

Liked this article? We have plenty more in store for you.

Subscribe to get news, tips and updates delivered to your inbox.

Read also

How To Optimize Email Subject Lines And Open Rates

Let's clarify the right subject line length once and for all and take a look at the details to pay attention to in order to ...

Read more

8 Essential Tips for Writing, Editing and Proofreading Your Emails for Powerful Results

Imagine a world where content writing for your business was as easy as stringing together a couple of sentences onto your computer screen and hitting ...

Read more

How To Streamline Teamwork, With Collaboration [Infographic]

When many heads work together in creating and approving an email campaign, the result can be a fine mess of comments and feedback. Thankfully, Collaboration ...

Read more

Great Drip Campaigns & Where To Find Them

Drip campaigns are simple and linear workflows triggered by user actions and behaviors: a fundamental tool in marketing and customer relations. Let's take a look ...

Read more

Email Preheader Best Practices

A guide to understanding email preheader - a small yet essential line of text that can boost open rates and consequently CTRs and conversions.  Email preheaders, ...

Read more

Email Checkup: Say Goodbye To Tiny Yet Fatal Technical Errors

One glance at checkup and you can discover the "health status" of your message, and identify how to remedy any critical issues. All this to ...

Read more