This isn’t the first time we’ve discussed the value of testing your emails on the blog. Best practices can get you started and help you develop your instincts, but ultimately that all goes out the door once your emails land in the hands of a customer. From there, it’s all about what the data shows you and whether or not they open, they click, and whether or not those campaigns are driving a sale.
With this in mind, here is an introduction to testing for your email marketing as well as a few ideas for things you should be testing.
A/B testing is amazing. It’s changed the face of online marketing, made software companies millions of dollars, and played a major role in presidential elections. But amidst the 100′s of blog posts extolling the virtues of testing and the countless articles on how to analyze tests, there has been far too little focus on what to test, and this has meant that A/B testing has been dramatically under-utilized.
The problem with how so many “growth hackers” use A/B testing is that they aren’t testing the big questions. Testing has come to focus so purely on optimization that it isn’t being used to define overall strategy.
Tests focused on helping unearth the right major strategic choices aren’t the type that have a 20% impact – they can have a completely redefining impact that might mean the difference in a company succeeding vs failing or increasing your growth rate by 10X. Optimization is really, really important – but it isn’t enough.
In the last year, I’ve received an average of 175 emails per day excluding what ends up in my spam folder. Assuming I spent an average of 30 seconds per email, that’s 90 minutes per day, 10.5 hours per week, and 22 days of my year. Give or take, this means I spent about 6% of my year on email.
While I only have the data to understand my email experience, pulling apart this number yields some anecdotal nuggets that might be useful in improving that 6% of my life that’s spent on email. If my email is at all representative, I’d propose we need a solution that both helps the receivers (who are consuming the email) and the senders (who are using email to send notifications, newsletters, etc).
Having previously found that the Obama campaign ended 1 in 6 of their email subject lines in colons, I decided to dive into the last 30 days of election emails (available online thanks to ProPublica) to see what they might tell me about Americans and how we respond to email more broadly.
From free car magnets to exactly which friends we should tell to vote, I realized that the emails we get are about to radically change as political campaigns and businesses get even more sophisticated at poking our psychological triggers and leveraging our social networks.
Do vs. Believe
I decided to create two word clouds out of the subject line to see how the language of each campaign differed. For each campaign, I took all of the email subject lines, took out the candidate names and extremely common words / combined words with the same meaning, and then sized each word based on how many times it was used.
Take a look at the word cloud at the top. Which campaign do you think it’s from? Now, look at this next word cloud:
The top cloud is President Obama, and the bottom is Mitt Romney. The most common words in Obama’s emails were share (or forward, like an email), calls and free – compared to America, help and love in Romney’s emails. Many of Romney’s email subjects wouldn’t have been out of place on motivational posters, while Obama’s would fit better as the names on Facebook buttons.
1. Jump on Facebook, 2. Make some Calls
Diving deeper, this change seemed to be reflected in the purposes of the emails sent. Nearly half (46%) of Obama’s emails explicitly asked the recipient to do something in the email subject line – whether that was to make phone calls, forward a message to friends, or sign-up for something – compared to 28% of emails.
Romney’s emails tended to focus on ideas:
- “A Strong America”
- “This is a time for greatness”
- “We Will Recover”
While Obama’s focused on taking action:
- “Three Things you can do right now”
- “Forward this:”
- “RIGHT NOW: President Obama Needs you to make some calls”
These emails represent strategic choices and differences in voter base certainly, but given the deep sophistication of the Obama campaign covered elsewhere, it’s very likely that giving people you email a direct next step in the subject is actually just more effective in many cases.
Tell Jason, Jim and Leah to Vote
One of the most interesting emails from Obama came on election day and had a personal list of your friends in swing states you should encourage to vote – all mined from your Facebook account if you’d given the campaign permission to access it. Based on other accounts, the Obama campaign knew who the voters on the line were in swing states – and based on these emails, it looks like they also knew which of their friends could encourage them to vote for Obama.
FREE Shipping (and $5 off!)
At varying points in the campaign, Obama was offering free t-shirts, free car magnets, free tickets to see James Taylor and Dave Matthews, free shipping – all to get people to donate. While the idea of free stuff to get people to do things isn’t new, the prevalence of these offers in the campaigns may mean that it’s poised to take over all the marketing emails we get.
3 Key Trends in the Future of Email Marketing
- More and more marketing will come through people we know. By getting people to engage their friends, marketers are able to increase their reach and give a personal touch to an otherwise impersonal ad or email. From mining Facebook data to tracking your demographics and actions around the web, marketers will know who you are, who you know, and what message will be most likely to resonate with you and your friends.
- Direct and clear next steps will become more pervasive. As social psychologists and marketers have long known, giving people an explicit action to take significantly increases the likelihood of them doing it. The Obama campaign emails suggests more and more advertising might include a direct call to action that is tailored explicitly to you.
- Marketing will become better – and harder to resist. As marketers become more sophisticated and metrics-driven, they’ll be able to stop sending you marketing that doesn’t have an impact. If I show that I like celebrities and contests, more of my emails may include free contests to hang out with celebrities.
If the 2008 election was the first election where donations of the masses played a bigger impact than the donations of the rich, then the 2012 election might be remembered as the election where email meant that even the concept of the masses has ceased to exist. We’re entering an age when political campaigns (and business) will send us personalized emails based on who we are and what we like – and they’ll ask us to bring our friends along with us.
Follow Klaviyo on Twitter for more analyses like this on the changing role of email marketing.
Early in 2012, I signed up for the Barack Obama and Mitt Romney campaigns’ email lists with a rarely used old email address. While I knew that this small dataset couldn’t reveal the extreme sophistication of their email strategies, I set out to analyze the emails I’d received (and rarely read) – and discovered some surprising differences in strategy (at least as it related to the emails I was sent).
The Emails in Numbers
From June 1st through November 5th, I got 35 and 37 emails from Mitt Romney and Barack Obama respectively (see chart above showing cumulative # of emails sent to me over time since June 1st).
There are two main differences you’ll notice – first, the sharp daily dose of Romney emails right after June 1st. Second, the month-long gap that immediately followed them. This gap began June 18th – the same day that I clicked the unsubscribe button on the Romney emails from another email address (once I realized I was signed up twice). While I can’t prove it, impressively, the Romney campaign seemed to realize I might be close to unsubscribing and put me on pause for a month.
Who Sends the Emails
When we break the emails down by who sent them, the results get interesting:
The Obama campaign is twice as likely to send emails from Barack Obama (49% of overall emails coming from him) than the Mitt Romney campaign is to send them from Mitt Romney (23% of overall emails coming from him). While the campaigns are roughly equal on the number of emails coming from Michelle Obama vs Ann Romney or Joe Biden vs Paul Ryan, there is a major difference in the use of others – a bucket largely made up of Zac Moffatt and Matt Rhoades (other Romney staffers) and his son Tagg.
A few hypotheses for why this might be true:
- A difference in strategy to add increased importance to emails from the candidate by sending fewer of them.
- Less candidate allegiance from Republicans in this election (and a greater emphasis on the party).
- Individual targeting or testing differences based on who I am. Had I exhibited some personal behavior that I liked emails from Obama but would prefer other people on the Romney campaign to Mitt? Is there someone out there who see the exact opposite of what I see?
The most interesting aspect of this finding is that it may reflect very real perceptions of what drives voters for each candidate – namely, more voters relating to Barack Obama on a personal level, and more potential Romney voters holding deeper party than candidate allegiances.
The One Word Subject Line
In a similar vein, while none of Romney’s emails had single word subject lines, about 1 in 7 of Obama’s did. Examples:
- Hey (this was a common one)
The one word subject line evokes a certain casualness and personal relationship and this difference seems to parallel many of the media portrayals of the candidates. Are the one word subject lines actually less effective for Romney? It’s hard to say, but what might be most clear is that the campaigns have developed real stylistic differences in how they talk to their constituents – and those could be rooted in the real differences of who their constituents typically are.
The Enigmatic Colon
Very unexpectedly, 1 in 6 of the Barack Obama message subject lines ended in colons (and none of the Romney subjects). Here are a couple of examples:
- Real Quick:
- This Matters:
Given how high this number is, my guess is that the Obama campaign has tested (and shown) that ending a message in a colon makes people more likely to read it. While the circumstances of a presidential campaign are obviously very unique, this isn’t a piece of advice I’ve heard elsewhere (and certainly not one that the Romney campaign has acted on in their emails to me).
The Future of Email may contain more Colons
First off, all of these analyses are based on a single person, and as ProPublica’s attempt to reverse engineer email strategies is starting to show, there are wide variations in what you’ll receive based on where you live, how old you are, whether you’ve donated, etc. As these systems get more complex, it will become more and more difficult to analyze any company or campaign’s email strategy – because that strategy might actually be 300 million different strategies.
That said, the emails will likely always say more about the particular cultures and moods of a campaign or organization at a given point in time. Would Obama letting Biden send more emails have changed how much money was raised? Would a “Hey:” from Mitt Romney have increased his chances of winning my vote?
The email strategies of the political campaigns are among the most sophisticated in the world and are a great indicator of how email will be changing as companies get better at linking the emails they send to the behavior of consumers. Just as Obama and Romney know what makes you press the donate button, companies are getting better every day at knowing how to make you purchase. In the future, it might not just be presidential candidates who are ending emails in colons and varying senders to figure out who you connect with – it might be your local farmers market stand.
Please tell us more about the Obama and Romney emails you’re receiving in the comments and if you want to know more about the future of email marketing, check out Klaviyo. And – go vote.
You are almost certainly losing money based on the emails you don’t send to your current customers. For some businesses, you are losing a lot of money.
From what I’ve seen, companies tend to fall into one of three buckets regarding customer lifecycle marketing emails:
- They don’t send them because they think customers will find them annoying.
- They don’t send them because they don’t think the value of setting them up will outweigh the cost in terms of time.
- They send too many and those have limited impact on customers.
These aren’t acceptable buckets to fall in. In each of these cases, they can know the answer by testing and then measuring what happens.
Here are 5 steps to help you get started:
Step 1: Implement something.
If you are reading this and haven’t put anything in place, stop immediately and go identify a set of customers and send them an email. One starting point is everyone who used to be very active but hasn’t been recently. If you’re worried about putting too much time into, focus on the simplest group possible and don’t worry about automating it.
Step 2: Pick a group as a baseline
In short, you want a group who lets you know what would happen if you didn’t implement the email you are planning on sending. For many of us, a great starting point is to just use what’s happened historically as this benchmark.
Over time, you probably want to consider holding out a set of customers as a control group who joined at the same time, from the same channels, etc. It’s not worth worrying about perfecting this at first.
Step 3: Measure Quantitatively
Once you’ve been sending emails, it’s time to see what’s happened. Two key things here:
- Focus on multiple metrics – sales, profit, site visits, unsubscribes, purchase size, etc. Even if a customer doesn’t change their buying or usage patterns immediately, you want to see if there are related positive (or negative) effects.
- Focus on impact over time, not just what’s immediate. Because it’s easier to track, too many companies focus just on the impact they see from clickthroughs, but you really care about what happens over the next 7, 14 or even 30 days.
The simplest way to do each of these is to look at average performance of customers at different times frames – 24 hours after email, 3 days, 7 days, 14 days, 30 days – of both the customers who received the email and those who didn’t. Over time, you’ll want to get more accurate and detailed about this, but this is a good starting point that will likely get you 80% of the right answer.
Step 4: Measure Qualitatively
It’s also worth identifying a random subset of customers and having quick conversations with them. The goal is to better understand the customer impact but also to generate new hypotheses about emails or approaches they might find helpful. Something to keep in mind is that what people say doesn’t always match with what they do. When Google initially tested many of their ads, they found that people reported not liking them when explicitly asked, but they also reported being happier on pages with ads than those without ads.
A final note – we have seen that offering a small token of appreciation (gift certificate, shirt, etc) helps engage people more quickly than otherwise. Depends on your business, but if you’re struggling with low response rates, might be worth considering.
Step 5: Automate
If it worked, go ahead and implement it. If you’re unsure, increase the size of the group or modify the email to try to create a test that you’ll be more certain about.
The main barrier is sending the first email. Measurement is a great way to figure out what’s working, but if you take it too far, it too can become a barrier to getting started. Good enough actually does tend to be a great place to start – once you know how your first efforts went, you can focus on optimization later.
The deep dark secret of nearly all analysis or metrics is that 99% of people will tell you they are a good idea, about 10% of people are actually looking at them, and about 10% of those are actually using them to significantly change their decision-making. This doesn’t change the fact that they are a great idea, it just means you need to be really careful not to waste your time.
Customer Lifetime Value
Customer Lifetime Value is a concept taught in countless marketing classrooms and is simple to grasp – it’s fundamentally the total value that a customer will spend with a business over their full life as a customer. What makes Customer Lifetime Value powerful is the realization that customers have relationships with businesses – so when we acquire a new customer, we’re actually acquiring all the purchases they’ll make with us over time. And the same goes when we lose them.
That sounds great – what’s the problem?
If you accept that Customer Lifetime Value is inherently a good idea (and the right way to think about things), you run into two problems when you try to use it to make decisions:
- It’s backward looking. Because you’re trying to figure out not what a customer’s value is today, but to predict what it’ll be over the next several years, you usually end up trying to factor in multiple years of past performance, or you end up trying to predict it off of a cohort analysis or regression model.
- It’s complicated. The reason we use metrics and analysis is to be able to evaluate decisions we are making – in basic form, you do X, you get Y. For this to work well, you need to have a ton of faith in Y and you need to be able to see it quickly. Because customer lifetime value is a prediction, we’re likely to spend more time trying to understand it, calculating it or debating it. Think back to how often analysis doesn’t get translated into action.
An Alternative Approach
Now, while trying to predict Customer Lifetime Value will remain crucial for some decisions, there’s an alternative approach:
- Try ideas often
- Observe the impact on clear metrics
- Measure those customers over time to understand long-term impacts
Let’s take an example. Say you are debating whether to send lapsed customers a discount offer to entice them to come back. If you take a subset of those customers and try it out, you’ll quickly know the answer to questions you could only speculate about before:
- How many customers come back? More than would have otherwise?
- How much do they spend? On what?
- Do they come back again over the coming months?
Customer Lifetime Value is an important reminder that we need to focus on the complete customer relationship over time – but as with all analytics, you need to make sure that your work gets translated into action.
Klaviyo lets you quickly implement and measure the impact of customer lifecycle marketing for your web app or ecommerce business. Try it today.