A/B Testing with Email at Cornell

We need to kick our email habit. It’s not easy to quit, and when each email brings more conversions, it’s tempting to just keep hitting send until you hit your goal. An easy way to solve this problem is to put a cap on the number of emails sent in a specific campaign, but audience size is harder to regulate. If your emails average a 20% conversion rate and you’re only allowed to send three, who’s stopping you from sending each of these three messages to 50,000 people?

At Cornell, we’ve thought a lot about how we send emails and have worked hard to adopt a mindset of only sending messages to those who are interested. This is easier said than done, as it’s impossible to completely predict what your audience wants to receive without them explicitly telling you. With behavior and interest data, you can get close. We decided to treat a person’s email behavior as their own expression of interest – opening an email as interest in a subject line or sender, clicking as interest in the content provided. Our team conducted an A/B test on the marketing of a suite of upcoming live streams using these indicators. Our control group all received the same set of invitations and event reminders if they had previously opened their event invitation while our experimental group only received event reminders if they had previously clicked on their event invitation. The experimental group attended events from follow-up emails at a rate more than ten times greater* than our control group.

These results absolutely confirm our suspicion that email behavior can accurately determine an audience’s interest in an offer. So why is it still so difficult to make the change to only sending follow-up emails to recipients who want them? Despite having an astronomically higher conversion rate from our follow-up messages, the experimental group still had about ten fewer attendees of each live stream event than the control group. In a business that focuses on getting as many people involved and engaged as possible, losing ten people can feel like too high of a cost. So how can we make it up?

Let’s first compare the email volume delivered to the audience that didn’t end up attending. In our interest-based group, only 69 people were sent an event reminder who didn’t attend compared to over 3000 in our control group. That number might not sound like it has a great effect on your email volume, but when you’re marketing hundreds of events a year, this habit can result in some serious email fatigue. When your audience gets tired of receiving your emails, they’re going to stop opening them.

Second – attendance can easily be made up by thoughtfully curating your audience. We use EverTrue to generate our audiences based on Facebook interest data and post engagement in order to market to people who we think would be more interested in what we are promoting. This means that those who receive the email are already more likely to open and click on your message in the first place. 

So here’s my proposal: send emails to only those who want them. Send follow-ups and event reminders to only those who have opened or clicked on your first message. Use your constituents’ demographics and web behavior to predict what they’re going to want to hear about. Changing your mindset and marketing strategy will move you closer to an audience-focused rather than an event-focused model. In an audience-focused campaign, marketers are pushed to consider not only the content of their messages in order to get audience members to engage but also pay more mind to who they are actually messaging. Our question can change from “How can I send my message to more people?” to “How can I get more people to open and respond to my message?” 

I’m not saying to throw out your conversion goals, but rather to change how you reach them. By only sending items of interest to your constituents, this decreases the number of emails from you in their inbox (goodbye, email fatigue!) and increases the value of each that they do receive. If we stop blindly throwing content-darts at our audiences, our focus shifts to what we can control: who we’re marketing to and what value we’re adding for them. 

* This messaging study was conducted on an audience of 21,415 Cornell alumni and friends. The audience members were selected based on past event attendance and Facebook post engagement (via EverTrue). 155 event reminders were sent to the experimental group and 6893 were sent to the control group. The average experimental and control conversion rates were 18% and 1% respectively, with a two-sided z test producing p<0.00001. We deemed a conversion as a recipient who clicked through a follow-up email and viewed the live stream for at least two minutes. Further testing is needed to determine long-term email fatigue between these two populations.

Author bio:

Emily Gustafson is the assistant director of digital marketing for alumni affairs and development at Cornell University. An alumna of the university, she now works within the division’s digital team to improve marketing performance through web tracking, data analysis, and creative problem-solving. Gustafson is a certified Google Analytics user and trained data visualizer.

0 0 votes
Article Rating
Notify of
Inline Feedbacks
View all comments