Trends in Mobile Test Automation
Test Automation and Unlimited Concurrency
First of all, it is difficult to perform a huge number of test runs on device farm without test automation.
Test automation is the enabler that makes concurrent use of mobile devices possible and enables users to run their application and test simultaneously on any number of devices.
This is also one of the sweet spots of any testing vendor/test team as there is no limitation on the number of devices that users can use simultaneously for their tests.
It’s a great news for the native app and game developers as they can use the same device roster too.
But, with slightly different test scripts and frameworks. The same foundation serves both the user groups extremely well.
Especially during the past two years, test automation services has come into play and helped mobile game/native app developers to quickly and easily see how their games work across device combinations and versions to get the vital debugging information on which things should be fixed before the release.
This has produced tons of great data of how game/ app can be optimized for end-user devices.
Today, the mobile test runs hammer thousands of devices every day, producing an enormous amount of data, results, screenshots, recorded videos out of devices runs, and performance statistics.
This shows how a mobile app/game work on these different handsets.
From the recent survey data, close to 84% of mobile test runs were done on Android devices and less than 16% on iOS devices.
This provided just a bit higher failure rate (17.8%) than on iOS devices (15.5%).
This sort of finding states that there are lots of issues also on iOS devices, with an increasing trend. Mainly, this is because of new iOS versions and some new form factors with iOS devices plus new APIs, new notification system etc.
What’s really awesome to see is that concurrency has been going up quarter by quarter. The average number of used mobile devices per test run is now over 10 devices per a test run. However, there is still lots of outliers and variation, as many of users still use one-device-at-time and some test automation experts use tens (or sometimes even hundreds) of devices simultaneously for their test runs.
The average test time also varies a lot.
For example
Quick and fast tests typically complete quickly to some extent, through Appium where some test runs may even take 10x time. As expected, it’s all about the scope of testing and the logics included in Appium tests.
The Most Popular Mobile Test Automation Frameworks
Appium is the most popular functional testing services of mobile apps, games and, to the certain extent, mobile web as well (we didn’t include Selenium in these numbers as the majority of all those selenium tweaks are more or less Appium-based).
There are lots of good reasons why people have been using Appium (cross-platform, support for literally any programming language, great coverage with API etc) and frankly, it has worked extremely well until recent changes with iOS UI Automation.
On Android, Espresso and UIAutomator have been very popular as well. And they’re also good reasons why people use/love these frameworks. Espresso provides very fast test execution and UIAutomator provides a light-weight API that is easy to adopt and use for your native apps.
Both these frameworks, however, are somewhat limited to native apps. Again, the majority of the developers either use Appium or some internal grown/developed framework.
Another (promising) new framework that has come into play is the Robot Framework and it quickly took over the Robotium, which used to be one of the mainstream frameworks in the early days of Android test automation.
On iOS Appium and Calabash, both being cross-platform frameworks, have been popular choices for iOS test automation.
Read more here: https://www.indiumsoftware.com/blog/trends-in-mobile-test-automation/