Problem 1: Our UI tests are stable, but we saw a lot of UI tests build failures. About ~50% of our CI builds were failing. All such failures of UI tests came from Spoon not being able to run tests on one or more emulators (device is red in the report and error message is …work/emulator-5554/result.json (No such file or directory), basically it timed out on installing the apk on a device, increasing adb timeout did not help, all emulators responded to adb commands and mouse/keyboard interactions, we suppose problem is in in ddmlib used by Spoon.
Solution: Composer does not use ddmlib and talks to emulators/devices by invoking adb binary.
Problem 2: Pretty often when test run finished, Spoon freezed on moving screenshots from one of the emulators/devices. Again, we blame ddmlib used in Spoon for that.
Solution: Composer invokes adb binary to pull files from emulators/devices, we haven't seen problems with that in more than 700 builds on CI.
Problem 3: Spoon pulled screenshots/files after finish of the whole test run on a device which slows down builds: test_run_time + pull_files_time.
Solution: Composer pulls screenshots/files reactively after each test which basically leads to: ~test_run_time.
Problem 4: If test sharding is enabled (which we do all the time), Spoon HTML report is very hard to look at, especially if you want to find some particular test(s) and it's not failed. You have to either hover mouse over each test to find out its name or go into html/xml source and find on which emulator/device test was sharded in order to click on correct device and then find test by CMD+F on the page.
Solution: HTML report we've built designed with usability and performance in mind.
Problem 5: Html report can be very slow to load if you have lots of screenshots (which we do) since it displays all the screenshots of tests that were run on a particular device on a single page — it can take up to minutes to finish while you effectively unable to scroll page since scroll is jumping up and down each time new screenshot loaded.
Solution: HTML report that we've built does not display screenshots on index and suite pages, screenshots are displayed only on the test page → fast page load.
With Composer we were able to make UI tests required part of CI for Pull Requests.
It's fast, reliable and uses RxJava which means that it's relatively easy to add more features combining complex async transformations.
HTML Report
Our Frontend Team helped us build HTML Report for the Composer.
It's fast, small and designed in collaboration with our QAs and Developers who actually use it on daily basis to make it easy to use.
Here are few screenshots:
Usage
Composer shipped as jar, to run it you need JVM 1.8+: java -jar composer-latest-version.jar options.
Supported options
Required
--apk
Either relative or absolute path to application apk that needs to be tested.
Example: --apk myapp.apk
--test-apk
Either relative or absolute path to apk with tests.
Example: --test-apk myapp-androidTest.apk
Optional
--help, -help, help, -h
Print help and exit.
--test-runner
Fully qualified name of test runner class you're using.
Default: automatically parsed from --test-apk's AndroidManifest.
Example: --test-runner com.example.TestRunner
--shard
Either true or false to enable/disable test sharding which statically shards tests between available devices/emulators.
Default: true.
Example: --shard false
--output-directory
Either relative or absolute path to directory for output: reports, files from devices and so on.
Default: composer-output in current working directory.
请发表评论