We talk a lot about A/B testing here on MarketingExperiments. What we don’t usually talk about is A/B testing for the mobile web…especially testing within mobile apps.
I thought we should change that. As I was scouring the web looking for mobile A/B tests, I found this 2-year old video by Amazon.
When they had testing, however, one app developer used it extensively and shared their experiences in a promotional video for the feature on Amazon. The developers were behind the game Air Patriots. Russell Caroll was the Senior Producer for the game and Julio Gorge was the Game Development Engineer. The game is a kind of aerial take on the classic tower defense game genre.
Granted, this was a promotional video, but the content still speaks for itself. These guys had (and still have by the looks of it) a fairly successful mobile app and they ran some successful tests. It’s a great starting place for what you can test in your mobile app.
By the way, while Amazon has shut down its A/B testing feature, there are many other tools for testing mobile apps that will accomplish the same thing the developers talk about in the video.
The team first tested the impact of ads on their customers. They wanted to make sure the ads did not harm the customer experience, so they tested a single ad in the main menu near the bottom of the screen.
They found that the ads didn’t affect customer retention. This meant they could insert ads and generate more revenue without hurting their customers.
In the second test, the team put ads in the game screen.
In both the first and second tests, a small “X” was placed in the ads that customers could tap to hypothetically dismiss them. When they tapped, a pop up came up telling customers they could eliminate ads with any purchase in the game’s store.
In this test, there was again, no impact on customer retention, but there was a statistically significant increase in revenue.
In this test, the team wanted to know if placing an icon to GameCircle (Amazon’s game stats and leaderboards portal) would improve performance.
It’s not clear which icon won, or even why this particular test was useful for the team, but they did get a favorable result. The lesson they wanted to drive home was that simple changes like icons can make a difference. We’ve also found that to be the case in a large number of our tests on MarketingExperiments.
In this fourth test, Caroll made a mistake. He accidentally changed the game difficulty to make it about 10% harder. As a result, every important metric the team measured tanked.
The team fixed it as quickly as possible, but it gave them an idea.
What would happen to revenue if they made the game easier?
So they ran a test with five treatments – the control and then four difficulty levels that were easier than the control.
It turned out that the easiest difficulty performed the best. Making it easier resulted in the players playing 20% longer and revenue increasing 20%.
The team then tested a push notification that offered inactive players and incentive for picking the game back up.
They wanted to know when the best time to send the notification would be. So they tested a few different variables and found that the best time was three days after the last game play.
They also found that sending the notification seven days after the last game play negatively impacted their performance metrics.
With these five tests and probably a few more taking place off the record, the team was able to develop a high-quality app for their customers and steadily increase their revenue. At the end of the video Carol gives a few key takeaways for marketers who are A/B testing their apps.
You might also like:
About Paul Cheney
Paul Cheney, Senior Partnership Content Manager, MECLABS Institute Paul helps turn raw research into easy-to-understand content for MarketingExperiments readers. He earned his B.A. in English literature from Covenant College. Before joining the MarketingExperiments team, Paul wrote grant proposals and fundraising letters for a mid-size nonprofit in New Jersey. He has also worked as a freelance Internet marketing consultant and copywriter for small businesses. In his spare time, Paul enjoys reading, writing poems and dating his wife, Callie.