What if A/B testing got you an extra $200M?

20 June 2013

I’d do a little testing for $200M. Wouldn’t you?

Two hundred million dollars. That’s the amount of money that Amelia Showalter attributes to the use of A/B testing during the Obama for America fundraising campaign.

Amelia is the former Director of Digital Analytics for Obama for America. She was interviewed earlier today by Daniel Burstein from marketingsherpa in a very interesting webinar about A/B testing. Lots of interesting lessons to learn. And that’s what this blog post is about.

AB testing success

The story of the Obama Campaign keeps getting studied and talked about. It has nothing to do with political preference. Quite simply, it’s the largest example of fundraising via digital marketing. And a successful one too!

  •  4.5 million donors
  • Over $500,000,000 in online donations
  • $53 average gift

That’s right. Email marketing was responsible for raising $500MM in donations. And as we mentioned, they estimate that the use of A/B testing was responsible for $200MM of that.

In other words, had they NOT tested different variations to pick the best messages to send each and every day, they estimate that they would have missed out on around $200MM in donations. Wow. How’s that as a strong statement for testing your email campaigns?

Conventional wisdom doesn’t help

Here are some of the truly interesting lessons that the team learned through hundreds of tests, in no particular order (for more details: you can find a PDF with slides from the webinar here).

  • Small improvements count! A 1% here and a 2% there, and all of a sudden you’re doing way better.
  • Send more, if you can: the “more email” policy drove $20 to $30M in additional revenue.
  • Send more, if you can: unsubscribes went up linearly … but donations did too!!
  • Short subjects, in general, tend to work better.
  • Conventional wisdom is often wrong… and so are established best practices. You need to figure out what works for you!
  • An ugly email might perform much better than a pretty one.
  • Incentives around a call-to-action work (e.g. free bumper stickers in some of their campaigns)
  • Since this stuff works, it makes sense to invest in human resources to focus on testing
  • Promote a culture of testing in your marketing team
  • Compare against your previous results: are you doing better than before?
  • Test all sorts of things: keep a testing calendar to plan what to test with which messages
  • The subject line is a big one to test (the biggest winner in their tests was “Hey”)
AB testing lessons learned

Does “Hey” work for geeks too?

The fact that “Hey” was a winning subject line is now a well known fact among email gurus. However, the folks at marketingsherpa – like many others marketers – were convinced that “Hey” would never work for them.

Of course, they couldn’t stop at what they “thought”. So they tried it, twice.

The results? They got much higher open and click through rates!!

The Obama campaign never chose a winner based on opens and clicks. The only metric that they used was donations. That’s all that counted. In the marketinsherpa test, they tracked opens and clicks, not conversions. So, it’s not “really” apples-to-apples, … but still, you get the point. They thought “Hey” as a subject line would only work for the President (or some other celebrity), and instead it appears to have worked wonder for geeky marketers too 🙂

A practical guide to A/B testing

Now that you’re sold on this A/B testing thing, here’s the next step for you.

AB testing ebook

At MailUp we’ve put together a dozen pages or so of practical advice on how to get started with A/B testing. It’s a quick ebook that can give you some fundamentals and get you started.

The ebook is divided into 5 parts, with a great preface by our friend – and email marketing guru – Marco Marini, CEO of ClickMail Marketing.

– Why A/B testing
– What to test
– How to run an A/B test
– Examples
– Check list

We used a provocative title for the ebook (“Keep picking losers? Send winner emails with A/B testing“), because – as the lessons learned above so clearly indicate, there’s really no way to consistently win unless you acknowledge that you need to test and learn.

Happy reading. And happy testing.

BTW: marketingsherpa are the folks that organize the Email Summit annual conference where we’ve exhibited twice already… and we’ll probably be there again next year. Great workshops for email marketers.

This article was written by

Massimo Arrigoni

Massimo Arrigoni

As head of products at MailUp, I spend my days (and nights) thinking of new tools that can help marketers get the job done more effectively. In MailUp 9, for instance, we just introduced a new way for our users to collaborate on email campaigns. With beefree.io, we took our drag-n-drop email editor and turned into a stand-alone, embeddable plugin. The objective is to create tangible value, gather feedback, and push real product innovation based on it.

Related articles