10 min

I’d do a little testing for $200M. Wouldn’t you?

Two hundred million dollars. That’s the amount of money that Amelia Showalter attributes to the use of A/B testing during the Obama for America fundraising campaign.

Amelia is the former Director of Digital Analytics for Obama for America. She was interviewed earlier today by Daniel Burstein from marketingsherpa in a very interesting webinar about A/B testing. Lots of interesting lessons to learn. And that’s what this blog post is about.

AB testing success

The story of the Obama Campaign keeps getting studied and talked about. It has nothing to do with political preference. Quite simply, it’s the largest example of fundraising via digital marketing. And a successful one too!

  •  4.5 million donors
  • Over $500,000,000 in online donations
  • $53 average gift

That’s right. Email marketing was responsible for raising $500MM in donations. And as we mentioned, they estimate that the use of A/B testing was responsible for $200MM of that.

In other words, had they NOT tested different variations to pick the best messages to send each and every day, they estimate that they would have missed out on around $200MM in donations. Wow. How’s that as a strong statement for testing your email campaigns?

Conventional wisdom doesn’t help

Here are some of the truly interesting lessons that the team learned through hundreds of tests, in no particular order (for more details: you can find a PDF with slides from the webinar here).

  • Small improvements count! A 1% here and a 2% there, and all of a sudden you’re doing way better.
  • Send more, if you can: the “more email” policy drove $20 to $30M in additional revenue.
  • Send more, if you can: unsubscribes went up linearly … but donations did too!!
  • Short subjects, in general, tend to work better.
  • Conventional wisdom is often wrong… and so are established best practices. You need to figure out what works for you!
  • An ugly email might perform much better than a pretty one.
  • Incentives around a call-to-action work (e.g. free bumper stickers in some of their campaigns)
  • Since this stuff works, it makes sense to invest in human resources to focus on testing
  • Promote a culture of testing in your marketing team
  • Compare against your previous results: are you doing better than before?
  • Test all sorts of things: keep a testing calendar to plan what to test with which messages
  • The subject line is a big one to test (the biggest winner in their tests was “Hey”)

AB testing lessons learned

Does “Hey” work for geeks too?

The fact that “Hey” was a winning subject line is now a well known fact among email gurus. However, the folks at marketingsherpa – like many others marketers – were convinced that “Hey” would never work for them.

Of course, they couldn’t stop at what they “thought”. So they tried it, twice.

The results? They got much higher open and click through rates!!

The Obama campaign never chose a winner based on opens and clicks. The only metric that they used was donations. That’s all that counted. In the marketinsherpa test, they tracked opens and clicks, not conversions. So, it’s not “really” apples-to-apples, … but still, you get the point. They thought “Hey” as a subject line would only work for the President (or some other celebrity), and instead it appears to have worked wonder for geeky marketers too 🙂

A practical guide to A/B testing

Now that you’re sold on this A/B testing thing, here’s the next step for you.

AB testing ebookAt MailUp we’ve put together a dozen pages or so of practical advice on how to get started with A/B testing. It’s a quick ebook that can give you some fundamentals and get you started.

The ebook is divided into 5 parts, with a great preface by our friend – and email marketing guru – Marco Marini, CEO of ClickMail Marketing.

– Why A/B testing
– What to test
– How to run an A/B test
– Examples
– Check list

We used a provocative title for the ebook (“Keep picking losers? Send winner emails with A/B testing“), because – as the lessons learned above so clearly indicate, there’s really no way to consistently win unless you acknowledge that you need to test and learn.

Happy reading. And happy testing.

BTW: marketingsherpa are the folks that organize the Email Summit annual conference where we’ve exhibited twice already… and we’ll probably be there again next year. Great workshops for email marketers.

Liked this article? We have plenty more in store for you.

Subscribe to get news, tips and updates delivered to your inbox.

Read also

The Statistics area is here: discover a new way of analyzing and improving engagement

Let us introduce a totally revamped reporting system. Beyond just a simple report collector, this active analysis tool measures sent and future campaign progress. The statistical ...

Read more

The best of 2019

Between joys and efforts, projects launched, and goals achieved—let's rewind the tape. Here's the (non-exhaustive) chronicle of what we've done to support your strategies. The year ...

Read more

Segment Your Campaigns In Detail: MailUp’s Advanced Filters Have Arrived

You can now create a filter by linking different conditions to segment recipients with maximum precision. The result? More flexibility, better engagement, and greater conversions. We've ...

Read more

Send Personalized Emails Based On Where Recipients Click

Now you can send automatic follow-up emails that are consistent with where a recipient clicks. This is quite a precious feature for leading prospects and ...

Read more

New Custom Rows! Edit, Order and Design Your Dynamic Content Blocks

Our custom rows are now all new and improved. Send out your latest blog articles or new e-commerce items dynamically - you can now create ...

Read more

Jade is Expanding: Now You Can Collect Contacts on Android Tablets Too

From today everyone, truly everyone, can even collect email addresses and phone numbers offline, directly from in-store tablets used at fairs and all kinds of ...

Read more