By Xiaojiang, from Idle Fish Technology
Flutter pages cannot use Native testing tools to locate elements directly, which brings a lot of inconvenience to automated testing. Although Google officially launched the Flutter driver and Integration test, the following issues exist in real-world usage:
Considering the issues above, we did not use the tools officially launched by Google but expanded the testing capabilities of Flutter pages based on the Native testing tools. This article analyzes the principle and implementation of the Flutter driver and Integration test and briefly introduces some schemes that Idle Fish has tried for automated UI testing.
When we first came into contact with Flutter automated testing, we tried to use the Appium framework to drive applications. When using the inspect function to dump page elements, we found that many elements would be merged into one area block and can only be located through XPath when clicking. It was difficult to locate some specific elements, and XPath was easy to change with poor code maintainability.
For the reasons above, we started to study the Flutter driver, an official testing tool provided by Flutter. When we first used the framework, we found that it can only be applied to pure Flutter applications but not to hybrid stack applications. However, the element locating capability provided by its bottom layer may be useful to us, so we analyzed its source code. The schematic diagram of the framework is shown below:
Figure 1: Schematic Diagram of Flutter Driver
The process interaction of the whole framework is relatively simple. When the testing script runs, it uses
FlutterDriver.connect() to connect with VMService to get the relevant isolate and then transmits the operation process and collects the data via WebSocket. All operations on the testing script side are serialized into JSON strings and passed to isolate through WebSocket. Then, they are converted to commands for execution on applications. For example, if we want to obtain the text content of a component, the final generated JSON struct is listed below:
After understanding the principles above, we can construct a protocol format to drive Flutter testing in any language or testing framework. Therefore, we encapsulate this protocol and use Python to drive it. By doing so, we can test Flutter pages based on UiAutomator2 and facebook-wda to meet the testing requirements of Flutter hybrid stack applications. The following is the final implementation code demo:
from flutter_driver.finder import FlutterFinder
from flutter_driver.flutter_driver import FlutterDriver
import uiautomator2 as u2
if __name__ == "__main__":
d = u2.connect()
driver = FlutterDriver(d)
if pageFlutter is True: # For Flutter，use Flutter driver.
finder = FlutterFinder.by_value_key("input")
We tried to use this framework. However, we found that the capabilities provided by the Flutter driver at the bottom layer are relatively weak and cannot fully meet our needs. The main problems are listed below:
As mentioned earlier, Flutter officially gave up maintaining the Flutter driver and introduced a new test framework called integration_test. Will this framework support hybrid stack applications? After trial, things were not as good as we thought. There is a sentence in the official document that says, "This software package can perform self-driving tests on Flutter code on devices and simulators."
The underlying element operations and locating of integration_test are still driven based on flutter_test. Its main advantages are listed below:
However, since element locating at the underlying layer is the same as the Flutter driver, the problems of the Flutter driver still exist. There are also other limitations:
Considering the problems above, it does not meet our requirements, so we did not conduct in-depth studies and application.
After learning about the relevant testing frameworks launched by Flutter official, we thought about how to design automated UI testing for Idle Fish. Should we develop new testing capabilities based on the official frameworks or reuse the existing native automated testing capabilities to expand Flutter testing capabilities? After considering the input cost and the maintenance difficulty of the testing script, we chose to use image processing technology to expand the support of the native automation framework for testing on Flutter pages. The architecture of the whole testing scheme is shown in Figure 2.
Figure 2: Architecture of Automated UI Testing Scheme for Idle Fish
Not all elements of Flutter are unrecognizable for UiAutomator2 and facebook-wda. So, when writing testing scripts, we only need to deal with the unrecognizable elements. We give priority to using the native locating capability to locate elements with name, label, and XPath that are not easy to change. We use image processing technology to locate other elements.
OCR text matching is preferred when processing elements that cannot be located using native capabilities. The accuracy is high and is not easily affected by resolution. Image search is used to locate pure images. We build a training set that uses image classification to determine the type of elements and locate common controls for some common element controls, such as commodity cards, prices, icons, and avatars.
The biggest problem facing automated UI testing is that as the version iterates, the testing script also needs to be iterated continuously. Therefore, the robustness and maintainability of the script need to be considered in the process of scheme selection and scripting. During script development, we encapsulate the page elements into separate classes and separate them from the testing logic, ensuring that only the corresponding page elements need to be modified during later element iteration. This reduces maintenance costs.
Figure 3: Script Hierarchy
The UI operations related to automated testing of Idle Fish performance have already used this scheme. There is no need to distinguish the type of current page when scripting. Our script has been running steadily over 500 times with a success rate of over 98%.
Figure 4: Scheme Comparison
As shown in Figure 4, the Flutter driver and integration_test are not mature enough to support hybrid stacks, but the Flutter driver can make some extensions. For pure Flutter applications, this scheme can virtually meet the testing requirements, while integration_test is relatively less mature. For testing hybrid stack applications, the scenario switching cost of hybrid stacks may still need to be considered. The OCR technology can be used for expansion, which may be cheaper and more profitable.
Thanks to SLM and TMQ for providing a lot of underlying capability support so that we can concentrate on our business.
XianYu Tech - August 10, 2021
XianYu Tech - August 10, 2021
XianYu Tech - November 22, 2021
Alibaba Clouder - September 21, 2020
Alibaba Clouder - December 22, 2020
Alibaba Clouder - February 8, 2021
Build superapps and corresponding ecosystems on a full-stack platformLearn More
Web App Service allows you to deploy, scale, adjust, and monitor applications in an easy, efficient, secure, and flexible manner.Learn More
Robotic Process Automation (RPA) allows you to automate repetitive tasks and integrate business rules and decisions into processes.Learn More
Explore Web Hosting solutions that can power your personal website or empower your online business.Learn More
More Posts by XianYu Tech