In the dynamic mobile gaming world, where user preferences and engagement patterns continually evolve, the importance of A/B testing cannot be overstated. A/B testing is the secret weapon that empowers game developers to make data-driven decisions, optimize player experiences, and fine-tune game elements for optimal performance.
By systematically comparing variations in gameplay, user interfaces, monetization strategies, and other crucial aspects, A/B testing allows game developers to deliver more engaging, enjoyable, and successful games.
It’s the essential tool that unlocks the door to understanding player behavior, leading to better retention rates, increased revenue, and, ultimately, mobile games’ continued growth and innovation. Below, we share examples of A/B tests and feature tests from our games supplemented with key takeaways you can apply to your games.
Makeup Kit Level Sorting A/B Test: Makeup Kit has a levels selection screen, where players can choose an image to create their makeup kits. The levels (e.g. images) are arranged in categories. Firstly, we analyzed the most popular categories. Next, we analyzed the most popular images. Using this data, we reorganized the popular categories and images to prioritize their appearance to find the best “sorting” result. This adjustment led to significant positive outcomes: D7 ARPU improved by 7%, D7 retention by 11%, and D7 playtime improved by 7%.
We can see how the images and categories were arranged on the left. On the right, the most popular categories and images are now displayed.
Key takeaway: This A/B test teaches us two things. Firstly, we should always try to collect data about player choices to identify the most effective strategy. Secondly, in games where the level order doesn’t impact gameplay, and players have a choice, regularly analyzing their preferences and adjusting the game accordingly will generally lead to improved outcomes.
Tie Dye Fashion Contest – A Meta Layer Test: In Tie Dye, we introduced a mock fashion contest (a meta layer feature) where players can design their tie-dye clothing and compete against other mock players. Players are even encouraged to rate other mock players’ clothes (e.g., tie-dye creations). This resulted in a 4.2% D7 ARPU improvement and a 6.4% increase in playtime.
The image on the left is the beginning of the contest. In the middle image, you (as the player) rate other mock players’ tie-dye creations. The contest results featuring all the mock players are displayed on the right.
Key Takeaways: There are two main points: 1) Incorporating mini-narratives can significantly enhance game engagement, especially in simulation games without “real” narratives. 2) The competitive element remains a powerful engagement factor, even when mock players are involved.
Frozen Honey – Offline Income Feature: In Frozen Honey, we experimented with a feature allowing players to select a waiter (either a cat or a dog), generating offline income. Players have the option to double this income by watching rewarded videos. By introducing this feature, we improved D7 ARPU by 9.6%.
Key takeaway: This test demonstrates that in games with effective “sinks” — useful ways for players to spend their in-game currency — introducing features that enable them to acquire more currency quickly stands a strong chance of succeeding. Moreover, these features are also effective for monetization of rewarded videos.
The image on the left showcases the feature where players choose their waiter, either a cat or a dog. On the right is the game’s main menu, with the offline income interface visible on the left side.
Mystical Mixing – Ad Profile A/B Test
We conducted a few A/B tests to determine the optimal ad profile. As a result, we increased the “Time between rewarded video and Interstitial parameter from 20 seconds to 30 seconds. This means that after watching a rewarded video, players won’t see any interstitial ads for at least 30 seconds instead of 20. This resulted in a 12.5% increase in D7 ARPU, a 7.5% increase in D7 playtime, 2% increase in D7 retention.
Key Takeaway: Typically, showing less ads results in lower ARPU. However, the goal of the A/B testing is to find the sweet spot between a high ARPU and delivering an engaging user experience. This is precisely why conducting A/B tests on these parameters is very important.
Makeup Kit – Introduce Coins feature Test: We aimed to conduct an A/B test for a new economy feature. Introducing a balanced economy in the game has the potential to enhance the ARPU by introducing more attractive rewards and expanding the availability of rewarded video locations.
The A/B test features “adding coins” as a reward upon completing levels. However, it’s important to provide players with something worthwhile to use these coins on, ensuring that the incentive to acquire coins remains strong. Our solution to this was leveraging one of the game’s most popular features, the Emoji Levels, and introducing a payment system using coins for their use.
The Emoji Levels allow players to pick four emojis and then select their makeup kit’s colors based on those choices. This modification not only expanded the potential for upcoming economy-related updates within the game but also led to a 3.5% increase in D7 ARPU, a 7.2% boost in D7 playtime, and a 5% enhancement in D7 retention.
Key takeaway: Establishing a well-balanced economy and providing players with incentives to spend is crucial. This test highlighted that sometimes (not always) requiring players to put in effort for something they desire can improve their overall experience and prolong their engagement.
By implementing coin-based access to the Emoji levels, players were motivated to “make an effort” for this goal, ultimately leading to prolonged playtime. However, this approach wouldn’t have succeeded if the reward – in this case, the Emoji levels – hadn’t been genuinely appealing. This is why you must ensure that the rewards you offer align with the value of players’ time and efforts.
In the image on the left, the game lacks a coin mechanic or any form of economy. In the right image, we can see the coins counter (in the upper right corner) and on the emoji levels button (where it says 225).
Partner With CrazyLabs on Your Next Game
CrazyLabs is a top 3 mobile game publisher and developer with over 6 billion downloads to date and over 250 million monthly active players in the hyper-casual, hybrid, and casual games space – it’s safe to say that we have the proven capabilities to make top-chart games. Let’s make your next game a hit.
