Whether you’re considering or already using email for your brand, acquisition, or retention efforts, email testing is vital to any marketing or communications strategy and will pay dividends in additional customer engagement and performance – no matter what your objectives are.
Email testing can be executed in a few different ways. We’ll look at A/B and Multivariate testing in this article plus provide you with some other ideas to get you started – it’s never too late to start email testing, no matter how small or large the idea!
What is A/B email testing?
A/B testing (also called split testing) is where you choose two elements that you test head to head, with half your email volume going into each. When you’re analyzing the results, it’s important to ensure that you have enough data to make a robust conclusion; the number of recipients you’re emailing will determine what results you can consider robust – the more the merrier!
You will also want to make sure that the metrics you are comparing are significantly different, for example, if version A had an Open Rate of 25% and version B an Open Rate of 26%, it probably wouldn’t be wise to consider a 1% difference significant enough to rule one version the winner, so you might want to test these again (unless of course, you decide just to choose the version you like best).
The Survey Star website has a nifty calculator to measure significance; simply enter the number of customers emailed in test A and test B, and the performance results that you’re comparing (open rate, click rate, click to open rate, conversion rate, etc.) and it’ll confirm whether the differences in performance are statistically significant.
What is Multivariate email testing?
Multivariate testing is where you have multiple variations being tested in different combinations at the same time, this can be a simple email test with as few as 8 variations, or as complex as 128+ variations. Be aware though that the more variations you have, the bigger your email list will need to be to accurately compare performance since you’ll be splitting your list into much smaller audiences.
Now, imagine you managed marketing at a pet supplies company and you wanted to test different email subject lines: one with a teaser (cryptic message to create intrigue) and one with the customer’s first name; and you also wanted to test which animal picture (in the email content) customers responded best to. Using multivariate testing you could mix and match these variations across 8 email combinations, as illustrated in the following table.
To calculate which of the two subject lines performed best, you can sum up all the variations which had the teaser, and separately all the variations which had the first name, and then compare the open rates. In this example, the teaser performed 6% better than using the customer’s first name.
When you measure the results of any email campaign, your open rates will indicate how your subject lines are performing, and your click rates will show you how the content in your email (headline, body copy, CTAs, images, etc.) are performing.
Now, let’s talk about the fun stuff – this is where you can be super creative and implement endless testing of your emails to keep your communications fresh and drive ever-improving results.
The following email testing ideas should help to get your creative juices flowing as you plan for future testing:
Subject line
When a customer receives your email, the subject line is one of the first things they see in their Inbox; in those first crucial fractions of a second, they use it to decide whether or not to open your email. There is a multitude of subject line tests, such as short and long subject lines, tone (humorous, informal, formal, etc.), you can pose a question, you can tease to create mystery or suspense, use title case, and personalization (such as including their first name or favorite product/movie/food, etc.) depending on data you have available (without being too “creepy”!).
Pre-header
The pre-header is the short text snippet that displays on some email clients, such as Gmail and Yahoo, immediately following the subject line in the inbox. It’s important not to forget this as it will default to the first line of text in your email, which might not be ideal.
The below example illustrates how this displays in Gmail and how it can go horribly wrong if overlooked. This is vital inbox real estate and can play a large role in whether a customer decides to open your email – so don’t let it default to “View in browser” or “Is this email not displaying correctly?”.
Format
How you decide to format your emails may depend on whether you have design resources readily available to you. Take a look in your inbox at emails you get from other businesses for inspiration or do a simple Google Images search. Depending on the characteristics of your customer base, you might like to test a plain text email that looks just like a personal one-to-one email without any imagery or fancy typography; or you might want to build something visually appealing with imagery, sections, animations, and video.
Hierarchy testing can tell you a lot about your customers and what they find important. For example, if you have multiple benefits for the product/service you’re promoting, you might choose to change/rotate the order that you communicate these benefits to determine whether customers respond better to some benefits more than others.
Whatever formatting you decide, don’t forget to add that very important unsubscribe link!
Copy
If you have varied data points on your customers, it can be beneficial to test varying copy to different customer types based on their demographics, past purchase or browsing behavior, location, etc. You’ll likely find different ways each customer segment prefers to be spoken to by comparing the opens, clicks, and conversions you receive from these copy methods.
There are endless ways to test email copy including, but not limited to:
- headline
- sub-headline
- copy length (short, medium, long)
- message type (soft sell, hard sell, product-based, benefits based, etc.)
- salutation (Hi, Hi Tom, Hi there, Hello, Dear Mr. Jones, etc.)
- sign-off (the friendly team at SMTP2GO, Until next time, Speak to you soon, etc.)
- tone (formal, conversational, humorous, etc.)
Emojis 😊
Emojis are a fun way of injecting some personality into your emails to better engage with your audience. That said, whether or not you use Emojis and which Emojis you decide to use, will entirely depend on your audience and the overall tone of your emails 🤔. You can test Emojis vs. No Emojis, different types of Emojis, placement of the Emojis 🕹 (in the Subject Line, body copy, or Call-to-Action for example) – the possibilities are almost endless! 🐙
Logo
Just like a plain text email can perform better than a well-designed HTML email, a logo can change how customers respond to your email. Therefore, it’s a good idea to test logo vs no logo, size of logo, and placement of logo to determine the best application for your customers and objectives.
Supplementary content
Supplementary content can be, well, just about anything you think is relevant or which your customers find interesting. This can be social buttons or excerpts, customer reviews, videos, blogs, or support articles. A word of caution though, use of such content should be determined by the primary objective of your email. For example, if the primary objective is to get your customers to click the CTA button and purchase a product, then supplementary content may distract and decrease overall conversions. That said, content can be a powerful way to drive awareness, inform and engage your audience; by improving how your audience perceives you, you can increase the likelihood of future purchase and referral.
CTA
Speaking of your primary objective, a well-placed CTA can do wonders for the performance of your emails. The key here is testing where these buttons/links are placed; how they’re worded and formatted can make a big difference in performance. For example, I once tested adding a greater-than sign “>” at the end of the CTA copy “Take a Look >” and the conversion rate increased 11% when everything else in the email remained unchanged.
Your CTA can be a button or a hyperlink; it can be placed once, twice, or three times throughout an email; it can be red, green, or blue (or whatever color works for you!). The key is to test CTA variations regularly to ensure that your emails are fresh, relevant, eye-catching, and delivers a response.
Imagery
They say a picture is worth a thousand words. In marketing and communications, an image can help illustrate, explain and convey an emotion or an action, among other benefits. When chosen carefully, in consideration of the audience and message that’s trying to be communicated, images can enhance how recipients engage with your emails. Knowing which images your audience will respond to is tough, and that’s why testing is a powerful tool to better understand your audience and their preferences.
Colors 🌈
Colors are powerful conjurors of emotion. They can subconsciously deliver a sense of calm, urgency, happiness, concern, optimism, and delight to your audience. Couple color with relevant messaging to create impact in your customer’s inbox. We have a great infographic here that will help you understand what colors trigger which emotions.
Time of day, day of week & follow-up ⏰
Testing when you deliver your emails will undoubtedly depend on the timeliness of your email; a supermarket wanting customers to visit on their way home from work might find better results if they send the email at 4 pm in the afternoon, whereas a bank wanting customers to provide feedback might find that their customers are more responsive and willing to volunteer 10 mins of their time at 10 am on a Saturday morning. Considering your audience and testing the time and day of the week that you send your emails is paramount to capturing customers’ attention – or risk being forgotten, unopened, or deleted.
Didn’t get noticed the first time? How about sending a second follow-up email? Do pay close attention to your unsubscribe rate as that’ll tell you whether your customers find your emails useful or just plain annoying.
Landing pages
The destination where your CTA lands customers is a crucial stage in the conversion process. Depending on your message and objective, a highly relevant landing page may translate to better performance.
Imagine you receive an email from a department store promoting a sale on t-shirts. You click on the email and land on their homepage – now you have to spend crucial seconds navigating to the t-shirts section. If you weren’t in dire need of a new t-shirt in the first place, it’s likely that you’ll lose interest. Now imagine you clicked through that email and landed on a page displaying the retailer’s great collection of t-shirts, one catches your eye – now you’re interested, right?
Email Testing with SMTP2GO
Email testing is offered to those on one of our Professional level plans (100K+ emails).
You may use this feature by visiting the “Reports > Email Testing” section of your SMTP2GO account’s dashboard.
From there, you can input your ‘From’ name and email address, email subject, and HTML code. Once you’ve created your test email, you can choose to “Preview in my inbox,” perform a “Spam filter test,” or do a “Screenshot test“.
Preview in my inbox sends a copy of the email to an email address of your choice so you can preview how it appears.
Spam filter tests let you see how major spam filters will handle your email. Depending on the spam filter, it may give you some more specific information as to any issues that may cause the email to be marked as spam. With this information, you can adjust the content of your email accordingly to ensure that your emails get delivered directly to the inbox.
Screenshot tests allow you to test how an email will appear to a recipient, within their given email client. You can choose from over 50 email clients across all of the major platforms for browser, desktop, and mobile-based email clients. When a screenshot test is run, we also run a link test on the given email. This will test all the links found within the email by checking whether the links resolve and if they are blacklisted or suspicious.
With endless ways to experiment with your emails and the customer journey beyond the email, the sky’s no limit for how you can start implementing small email tests each and every time you communicate with your customers.
Good luck out there! 🍀
