A/B Test to Stay Ahead of the Competition
Anyone that runs a business will agree that competitive analysis is essential, but it’s also easy to get caught up in thinking, “Well, my competitor is doing XX. We need to do it too!” However, what works for competitors might not work for you (remember Zune?).
This is where A/B testing came in for Paktor, a dating app targeted at the Asian market. The app uses a thumbs up icon for users to indicate interest in a potential match. However, other dating apps use a variety of icons for the same purpose including hearts, checks, and message icons. The Paktor team discussed many different ways to improve engagement (like rate and repeat app usage rate primarily). They were fairly certain that changing the thumbs up icon to a heart icon would improve engagement rates. Not only does it in theory better reflect the love seeking purpose of the app, many competitors seem to have had success with heart icons.
Original App With heart icon
As online dating grows, the selection of dating apps is increasing. Yet regardless of the app, the core concept of expressing interest in a potential match is universal to all digital dating. In every dating app, there is some way for the user to tell the app, “yes, I like this person” or “yes, I want to get to know this person.” But the icon that users click to express interest is different from app to app.
Coffee Meets Bagel Tinder Match.com
This is arguably the single most important icon in a dating app. The app can only make connections if users click on this icon. This means that the app only has value and remains engaging if this icon is clicked. Regardless of the dating app’s monetization strategy, a large pool of engaged, repeat users is the key.
Paktor wondered which icon would create the most engagement. They hypothesized that the a heart would better represent finding love, so they A/B tested an x and heart icon combo against the thumbs down and thumbs up icons the app originally had.
Original icons New A/B Test
Thankfully they A/B tested it because the x and heart combination lowered engagement by 6% and would have cost them 26% in month over month growth.
Despite the fact that the x and heart icons seem to be working for other dating apps like Tinder, Clover, and Hinge, Paktor believes that the heart icon signals a level of commitment the Asian user is not comfortable with.
The heart icon reduced repeat uses by a statistically significant 2.3%. That’s a loss of 11% in revenue over a year.
Paktor was ready to implement the change without testing. That’s how certain they were that the x and heart icons would be better. The lesson they learned was that every hypothesis is worth testing.
About the author: Lynn Wang is the head of marketing at Apptimize, an A/B testing platform for iOS and Android apps designed for mobile product managers and developers alike. Apptimize features a visual editor that enables real-time A/B testing without needing app store approvals and a programmatic interface that allows developers to test anything they can code. Sign up for Apptimize for free today or read more about mobile A/B testing on their blog.
