Selection and Exploration of Scheme for Flutter Automated UI Testing

This article analyzes the principle and implementation of the Flutter driver and Integration test and introduces schemes that Idle Fish has tried for automated UI testing.

By Xiaojiang, from Idle Fish Technology

Flutter pages cannot use Native testing tools to locate elements directly, which brings a lot of inconvenience to automated testing. Although Google officially launched the Flutter driver and Integration test, the following issues exist in real-world usage:

It does not apply to hybrid stack apps. There are related drivers in Appium, but we cannot switch environments.
The element locating ability is relatively weak.
It depends on VMService, and we need to build a Profile or Debug packet.

Considering the issues above, we did not use the tools officially launched by Google but expanded the testing capabilities of Flutter pages based on the Native testing tools. This article analyzes the principle and implementation of the Flutter driver and Integration test and briefly introduces some schemes that Idle Fish has tried for automated UI testing.

1. Flutter Driver

When we first came into contact with Flutter automated testing, we tried to use the Appium framework to drive applications. When using the inspect function to dump page elements, we found that many elements would be merged into one area block and can only be located through XPath when clicking. It was difficult to locate some specific elements, and XPath was easy to change with poor code maintainability.

For the reasons above, we started to study the Flutter driver, an official testing tool provided by Flutter. When we first used the framework, we found that it can only be applied to pure Flutter applications but not to hybrid stack applications. However, the element locating capability provided by its bottom layer may be useful to us, so we analyzed its source code. The schematic diagram of the framework is shown below:

Figure 1: Schematic Diagram of Flutter Driver

The process interaction of the whole framework is relatively simple. When the testing script runs, it uses FlutterDriver.connect() to connect with VMService to get the relevant isolate and then transmits the operation process and collects the data via WebSocket. All operations on the testing script side are serialized into JSON strings and passed to isolate through WebSocket. Then, they are converted to commands for execution on applications. For example, if we want to obtain the text content of a component, the final generated JSON struct is listed below:

{
    "jsonrpc":"2.0",
    "id":5,
    "method":"ext.flutter.driver",
    "params":{
        "finderType":"ByValueKey",
        "keyValueString":"counter",
        "keyValueType":"String",
        "command":"get_text",
        "isolateId":"isolates/4374098363448227"
    }
}

After understanding the principles above, we can construct a protocol format to drive Flutter testing in any language or testing framework. Therefore, we encapsulate this protocol and use Python to drive it. By doing so, we can test Flutter pages based on UiAutomator2 and facebook-wda to meet the testing requirements of Flutter hybrid stack applications. The following is the final implementation code demo:

from flutter_driver.finder import FlutterFinder
from flutter_driver.flutter_driver import FlutterDriver
import uiautomator2 as u2

if __name__ == "__main__":
    d = u2.connect()
    driver = FlutterDriver(d)
    if pageFlutter is True:  # For Flutter，use Flutter driver.
        driver.connect("com.it592.flutter_app")
        finder = FlutterFinder.by_value_key("input")
        driver.tap(finder)
        time.sleep(1)
        print(driver.getText(FlutterFinder.by_value_key("counter")))
    else:
        d(text="increase").click()

We tried to use this framework. However, we found that the capabilities provided by the Flutter driver at the bottom layer are relatively weak and cannot fully meet our needs. The main problems are listed below:

Elements cannot be manipulated in batches. An exception is returned once the finder has located more than one element.
Most of the time, developers do not write keys, and element locating is not convenient.
Since Flutter has no inspect tool to dump elements, it can only use the source code to write scripts. Thus, the code maintenance cost is relatively high.
Flutter has stop maintaining the project, so it is estimated that there will be no new functions.

2. Integration_test

As mentioned earlier, Flutter officially gave up maintaining the Flutter driver and introduced a new test framework called integration_test. Will this framework support hybrid stack applications? After trial, things were not as good as we thought. There is a sentence in the official document that says, "This software package can perform self-driving tests on Flutter code on devices and simulators."

The underlying element operations and locating of integration_test are still driven based on flutter_test. Its main advantages are listed below:

The testing script can use all Flutter APIs.
After packaging IPA and APK, users can run tests on Firebase Test Lab and other device groups without additional drivers.
There is no correlation between tests on each page of integration_test, and tests for a single page can be implemented.

However, since element locating at the underlying layer is the same as the Flutter driver, the problems of the Flutter driver still exist. There are also other limitations:

The testing script is packaged into the applications. Each time the script is modified, it needs to be repackaged.
It is not friendly enough for end-to-end testing and requires additional functions to wait for data to be loaded.
It is not suitable for page testing at the full procedure level.
It has weak scalability.

Considering the problems above, it does not meet our requirements, so we did not conduct in-depth studies and application.

3. Automated UI Testing Scheme for Idle Fish

After learning about the relevant testing frameworks launched by Flutter official, we thought about how to design automated UI testing for Idle Fish. Should we develop new testing capabilities based on the official frameworks or reuse the existing native automated testing capabilities to expand Flutter testing capabilities? After considering the input cost and the maintenance difficulty of the testing script, we chose to use image processing technology to expand the support of the native automation framework for testing on Flutter pages. The architecture of the whole testing scheme is shown in Figure 2.

Figure 2: Architecture of Automated UI Testing Scheme for Idle Fish

Not all elements of Flutter are unrecognizable for UiAutomator2 and facebook-wda. So, when writing testing scripts, we only need to deal with the unrecognizable elements. We give priority to using the native locating capability to locate elements with name, label, and XPath that are not easy to change. We use image processing technology to locate other elements.

OCR text matching is preferred when processing elements that cannot be located using native capabilities. The accuracy is high and is not easily affected by resolution. Image search is used to locate pure images. We build a training set that uses image classification to determine the type of elements and locate common controls for some common element controls, such as commodity cards, prices, icons, and avatars.

The biggest problem facing automated UI testing is that as the version iterates, the testing script also needs to be iterated continuously. Therefore, the robustness and maintainability of the script need to be considered in the process of scheme selection and scripting. During script development, we encapsulate the page elements into separate classes and separate them from the testing logic, ensuring that only the corresponding page elements need to be modified during later element iteration. This reduces maintenance costs.

Figure 3: Script Hierarchy

The UI operations related to automated testing of Idle Fish performance have already used this scheme. There is no need to distinguish the type of current page when scripting. Our script has been running steadily over 500 times with a success rate of over 98%.

4. Summary

Figure 4: Scheme Comparison

As shown in Figure 4, the Flutter driver and integration_test are not mature enough to support hybrid stacks, but the Flutter driver can make some extensions. For pure Flutter applications, this scheme can virtually meet the testing requirements, while integration_test is relatively less mature. For testing hybrid stack applications, the scenario switching cost of hybrid stacks may still need to be considered. The OCR technology can be used for expansion, which may be cheaper and more profitable.

5. Acknowledgments

Thanks to SLM and TMQ for providing a lot of underlying capability support so that we can concentrate on our business.

Community

Selection and Exploration of Scheme for Flutter Automated UI Testing

1. Flutter Driver

2. Integration_test

3. Automated UI Testing Scheme for Idle Fish

4. Summary

5. Acknowledgments

Read previous post:

Read next post:

XianYu Tech

You may also like

Comments

XianYu Tech

Related Products

EMAS Superapp

Web App Service

Robotic Process Automation (RPA)

Web Hosting Solution