There’s no 100% approval rating, but you can get close!
A/B testing, for most of us, means “always be testing.” And it’s true that constantly measuring how changing one aspect of your push notifications can give you valuable data about your users and what motivates them. But how do you pick the As and the Bs? Here’s a practical guide to the best ways to run A/B testing.
The As And The Bs
Remember, the goal of A/B testing is to measure the effect of one change, whether you swap out the action in a notification, change a word of the copy, or some other small tweak. Your A group is what a scientist running this would call a “control.” That’s a group where nothing has changed; you treat them as you normally would. The Bs are the group who get the change. The idea is that you compare the two and their reactions to see what, if anything, changes.
First, you need to start with an overall population; that is, a set of users who have received a notification from you already. For ease of use, let’s say that you’ve pushed out a global notification to everyone on your opt-in list, like an announcement of a new podcast about your industry, and that announcement directs them to a page to learn more. Your metric is how many subscriptions to the new podcast you get.
So, you push out the message, and you’ve got a sense of the conversion rate. First, if you can sort out those who’ve already subscribed, do it. Now, you’re going to make the change wherein the A group, they’re directed to the pitch page, but the B group is taken to another page with the first episode embedded and subscription links. But who’s a B?
Who’s going to volunteer?
There are two generally accepted methods for picking out Bs. The first is a simple “every-other.” You generate a master list of who you’re pushing to, and every other person is slotted into the B group. This has the advantage of being direct and easy to execute.
A drawback, though, is that it’s not purely random and it’s a bit unfocused. Each customer is unique and is going to belong to different segments. For really big lists, or really small lists, it’s not a bad approach, but that lack of focus may lead to “noisy” data it’s hard to pick trends out of. A second is to pick out two segments and treat one as A and one as B. This works if you want to be exceptionally focused, but it will only tell you about those two segments.
Finally, there’s what a scientist would do: Fully random, or at least as random as possible. This works best when you’re not really interested in specific segments, but the global whole, and that’s what we’d use for our podcast test, in this case.
Which method you use will depend on what you want to know and how tightly focused you want to be. So don’t hesitate to vary it up as you test, and see what gets you the best data. Ready to see how push notifications can build loyalty with your best customers?
Sign up for a live demo of Pushnami!
