Experimentation as the defining trait of growth
The experimentation process is around choosing that goal,and then defining a series of experiments that you can use to achieve that goal and gather learnings along the way.
Grow from 50% to 60% on month-on-month retention
Say, for example, you're a growth marketer at Instacart, and you are responsible for increasing retention rates of existing users at Instacart. You had a quarter to come up with a plan and run a series of experiments in order to move the metric from, say, 50% month-on-month retention to 60% month-on-month retention.
Traditional Marketing - run an email campaign
Traditional brand marketing - they probably wouldn't be focused on this metric. The approach would be run an email campaign to these users reminding them to use Instacart.
Don't set it up as an A/B test. Just send them an email. See, if after they get the email, people buy more groceries or come back to the site more often. You can do that a couple times over the course of the quarter and each time you send an email you can see is there a lift in retention rate,or does it drive traffic to the website.
But what you're not learning there is whether or not the message that you're sending or the email your sending is the right message or the best message,or exactly how much lift are you getting and retention from sending this email. Maybe it's actually not being caused by the email. Maybe they're getting this email, and then there happens to be something else that's going on that's causing an overall increase in retention rates. Maybe it rained that week and people are particularly lazy and are ordering more from Instacart.
Run 5 a 10 experiments to learn
If you don't run as an experiment, you don't actually know is the campaign that you ran, is the experiment that you ran the thing that is driving this improved performance of this metric,or is it something else entirely. So what I would do is I would come up with a series of five, 10 experiments that I want to run that I think would increase retention rates.
Use hypothesis - Is sending an email an increasing of the retention rate ?
The way you do that is you come up with a hypothesis. Say, if people get reminded that Instacart is convenient, they actually use this email, they send it to me and they say,"Save yourself an hour this week. "Use Instacart instead of going to the grocery store. "Set it up as an A/B test. See if the people that receive the email actually buy more groceries or come back to the site more often than people that don't see the email.
Then you learn does the email itself actually increase retention rate or purchase rate of customers versus not getting the email at all.Then if you learn that sending emails is a good idea, and actually increases purchases, you can then move on to the next hypothesis, which could be what is the right message to send people?
Testing your message.
If sending emails is good, how can I send them the best email that's going to lead to the most purchases and the most retention?
So that's when you start testing messaging. You say maybe it's about convenience and saving time. Maybe it's about pricing. Be like, "You will save 20% by using Instacart", or some random thing like that. Then you can learn do customers care more about the convenience factor or do they care more about price? Then by setting it us as an experiment you know exactly based on purchase rates and retention rates of the convenience test group versus the pricing test group which message is more compelling. Potentially, you can go even a layer deeper and learn which customers exactly care more about pricing or convenience, because realistically, not everybody cares about the same thing.
Finding the right message, the right offer, the right customer experience for each individual customer
So at the end of the day, the ideal state of any growth processis to get to the point where you know what is the right message, the right offer, the right customer experience for each individual customer in your business in order to drive the best experience and the best results.
Three layers of depth
First layer - check of an email campaign is working ?
There are essentially three layers of depth that you can go to with an experiment. The first is just learning whether or not doing an email campaign or a push notification or a landing page at all has some sort of affect on conversion. You can learn that sending an email improves conversion. That's great. Your competitors are probably doing the same thing.
Second layer - which is the best message ?
So if you want to go a layer deeper, it's about figuring out what is the right message or the right offer or the right campaign? How do you actually get better about your email marketing? Or how do you make better landing pages? So for your entire audience,then how do you create the best experienceto drive 10% more growth or 20% more growth on top of the most basic? It could be the worst email campaign possible, but sending an email, even if it's terrible, is better than no email. So how do you come up with the best email?
Third layer - which customers respond best to which type of offer
Then once you figure that out, or at least once you start seeing some gains and figuring out there are certain messages that work better than others, the third layer, and this is where things get really powerful,is when you can actually tailor the message to individual customers. Understand what campaign, what offer, what message resonates with customer A better than with customer B. Because trying to treat all your customers the same, assuming they care about the same things, they like your brand for the same reason,or they use your products in the same way is wrong.I can guarantee you it's wrong. It is possible now with the data that companies have available to them by running experiments you can learn exactly which customers respond best to which type of offer or message. So you can probably get to 10 or 20% improvement by just testing different messages or different campaigns against each other at the macro level.
But once you start getting into every individual customer has different experiments run for them, so you're learning what is the best thing for each individual,that's when you get to the point where you're at 30, 40, 50% better than your current baseline.
That's where you can differentiate versus your competition, because it's not that easy to do. There are a lot of companies that don't do it. The best companies do this religiously, and they probably don't want other people to know about it because it is so powerful. It takes a lot of time and energy to invest in and get good at. But once you figure out how to do it and you have it instrumented correctly,the results are incredibly powerful, because every person is unique and wants different things.If you can give that to them, you'll not only improve metrics in the near term and you'll see better performance, but you will also see longer term benefits .
People will be more loyal to your brand. They'll feel that one-to-one personal connection when you can deliver them personalized messages and optimize everything for the individual. So that's the holy grail of experimentation and growth.
An example - let's talk about food delivery now.
You've got Postmates, DoorDash, Uber Eats, Seamless, all of these different, essentially, commodities. They're all doing some sort of email marketing experimentation, testing different landing pages. If you're all doing the same thing, if you even get to the best single message for your entire audience, say, you get 10 or 20% better than your baseline, you're still not necessarily better than them. You're definitely better than where you were before and that's great.Improving your monthly email resurrection or retention email by 10 or 20% if your average purchase value is, I don't know, $20 or $30 per customer, and you've got millions of customers, you're talking about hundreds of thousands of dollars in incremental revenue every single month.
As your customer base grows, it's $100,000 this month, and then it's $120,000 the next month, and then $140,000, and then by the end of the year this 10% increase in revenue per user from a better email that started out as $100,000 more a monthis now $400,000 more per month if you're growing your business.
Buy yeah, but then if you can get to the next level, every incremental bit of performance or conversion performance you improve translates to down funnel effects for your business in the long run. So it's great to get that first incremental $100,000 in that first month. But when you actually extrapolate that to over the course of an entire year as your overall business is growing, when you move one of these metricsit turns into something that continually compounds and adds more revenue for your business over the long run.
Try and fail
If you set something up as as experiment, even when you fail, you are learning. Therefore, it is valuable. As opposed to if you're not running as an experimentand you're just doing stuff, you're just sending emails, you're just putting up landing pages, if it's not an experiment and you're not learning does it work, or does it not work, you're not getting any learning as about what your customers care about, what things are working, what things aren't working. When it fails, and you learn about something that doesn't work, that's just as valuable. That's one more data point that you can use to improve every other future thing that you do in your business.
Save money by a failed experiment
A lot of times, those benefits actually aren't even limited to your team. If you run an experiment around messaging, and you learn that customers don't care about, I don't know, saving 10 minutes by ordering from Instacart versus going to the grocery store, whatever the example is, and your experiment failed, you didn't see any increase in conversion, you're like, well, darn this was a failed experiment. Then you realize that you just eliminated one messaging point that you probably would have used again,that you didn't know wasn't working. There are probably other people in the business that are using similar language that are unintentionally failing and doing a bad job at communicating your value proposition to customers. You just saved all of those other people time, and likely money, that they're losing, because they were using the wrong message.
Half the money you spend on advertising is going to be wasted. You just don't know which half. It's fine if you waste that much money, but once you learn which part of it is wasted, that's how you can start fixing that and getting better. It's the thing that makes failure into a learning opportunity, which is extremely valuable. Because that means even as you experiment more, as long as you're getting some things right, even one out of every four, then you're doing pretty well, and you're only going to get better,and your success rate's only going to increase over time.
