Gordon
Assistant Engineer
Assistant Engineer
  • UID622
  • Fans1
  • Follows0
  • Posts52
Reads:1107Replies:0

Freeline - second-level compilation solution on the Android platform (1)

Created#
More Posted time:Dec 6, 2016 9:37 AM
Freeline
Demystify Freeline technology
What is Freeline?
Freeline is a dynamic-replacement-based compilation solution that Ant Fortune team, a one-stop financial management platform under Ant Financial, developed for the Android platform in October 2015. Freeline went open-source inside Alibaba Group in May. In terms of stability: Freeline features complete baseline alignment, and process-level exception isolation mechanisms. In terms of performance: Freeline is inspired by the concurrency ideology for multiple projects and multiple tasks of the open-source tool Facebook Buck, and adopts internal policies such as port scanning, code scanning, concurrent compilation, concurrent DX, and concurrent merge DEX, boasting a significant acceleration effect on multi-core servers. In addition, Freeline implements corresponding cache policies at the class, DEX and resource levels to achieve true incremental development. It also introduces and optimizes some acceleration components of Facebook Buck such as DX, DexMerger and resource compiling, and makes in-depth renovation on the resource compilation process of AAPT. Thanks to this, when the resource changes, the incremental package compilation can be completed within seconds. In particular, the incremental package only contains the smallest set of changes (with the size ranging from 10 KB to hundreds of KB) and is also used online for dynamic replacement of resources/code. As a result, Freeline is several times faster than the current Instant-Run, Buck, and LayoutCast solutions.

What advantages does Freeline have?
1. Truly incremental, boasting fast builds and a small incremental package size greatly improves the speed for deploying changed code to the mobile phones, namely three to five times faster than Android Studio 2.0 and LayoutCast.
2. Cross-platform support of Linux, Mac OS and Windows.
3. Full coverage of various versions from 2.x to 6.x.
4. Simple deployment processes. After the code is changed, a TCP persistent connection will be established with the mobile phone in the build process, so that one line of command can complete the increment deployment, saving the entire process for building the package in the directories of various child bundles, packaging the increments in portal/launcher and installing the packages to the mobile phone.
5. Support of transactions. The exceptions introduced during the development process will not impair the workspace.
6. Seamless support of MPaaS, solving the inconsistency with general development processes, such as the merging of various Maven nodes.
7. Process-level exception isolation to ensure the continuous stability of development experience.

Who are using Freeline?
Currently, Freeline offers stable support to the daily development of Android technical teams such as Ant Fortune and amap.com and is compatible with the MPaaS/Gradle architecture.
Open-source address:
https://github.com/alibaba/freeline

Technical inside story: comparison of solutions
Let's first look at the conventional Android packaging process:

The conventional packaging process in Android is conducted in a single process following a pipeline of tasks from top to bottom for packaging and builds. In the process, the AAPT (Android Asset Packaging Tool) will be executed twice, first for generating R.java for javac compilation, and second for compiling the resource files in the res file. At last, the APKBuilder will package the DEX file and compiled resource files into an APK, sign the APK and install it to the mobile phone. There is no caching, no concurrency and no increments throughout the entire process. Every package build is a fresh new process, so the time used for every build is also consistent. The larger the code volume and resource volume, the longer the build takes.

Next we will make comparisons between and analyze the three mainstream incremental build solutions in the industry:
LayoutCast:
LayoutCast is the originator of the increment ideology. Unlike LayoutCast, Freeline breaks down the scanning and incremental package build tasks that connect the device and various projects by referring to the practices of Facebook Buck, while LayoutCast adopts the conventional pipelines of build tasks because of which the performance may be compromised by some prolonged intermediate procedure and the multi-core advantage is not fully leveraged. In addition, LayoutCast upholds the same idea for selecting the solution for resource changes with the Instant-Run of Android Studio 2.0. It packages the resources of the whole application into a resource package and pushes it to the mobile phone. If the resource package is big in size, the process will consume a considerable length of time, and the size of the resource package directly influences the subsequent TCP transmission duration. From the perspective of resource changes, LayoutCast fails to implement increments in the true sense.
In addition, LayoutCast generates ids.xml and public.xml by reflecting the R Class field during the runtime on the mobile phone end to ensure the consistency between the resource IDs of the incremental package and the full package. But this solution has a few defects:
1. Reflection on the mobile phone end during runtime involves tens of thousands of R class fields, with a very low efficiency. For example, this process may take nearly 1 second on Galaxy Note 4.
2. A fatal defect exists. For example, if the attrs.xml that the app declares has a definition as below:
<attr name="ptrMode">
       <flag name="pullFromStart" value="0x1" />
        <flag name="pullFromEnd" value="0x2" />
 </attr>


The ID values corresponding to “pullFromStart” and “pullFromEnd” are as follows:
public static final class id {
        public static final int pullFromEnd=0x56050005;
        public static final int pullFromStart=0x56050004;
   }


In fact, the ID type generated by the above two enumeration constants is “id”. If these enumeration IDs are not eliminated when ids.xml and public.xml are generated, the final result will be an array out-of-bounds exception and AAPT core dump when AAPT allocates the ID to every resource, resulting in resource package build failure. What's more, during the runtime reflection on the mobile phone end, the information of ID type of “pullFromStart” and “pullFromEnd” is not enough to deduce that they correspond to enumeration constants. Therefore, runtime reflection alone is impossible to solve the above fatal defect.
Furthermore, the absence of the caching mechanism makes LayoutCast compilation slower and slower with the increase of modified files. Finally, to implement code increments, the incremental DEX is inserted to the beginning of the dexlist of the system and this approach will not pass the system security validation on Android 4.x devices, which is why LayoutCast does not support mobile phones lower than Android 5.0.

Buck
Next let's talk about Buck. Here is its official build process chart:


Buck breaks down the tasks on a single pipeline into multiple child task nodes for concurrent execution by project. It streamlines the dependent relations between the nodes and summarizes a directed topology to build various child task nodes through multiple concurrent threads, fully leveraging the multi-core advantage.
Buck establishes a complete set of sound dependency rules and a detailed caching system to shorten compilation time. The principle of its incremental build is actually to build increments with the project directory as the unit. When a change occurs, the changed project, and the projects taking the changed project as the parent node or ancestor node all require re-building. After these projects involved in the change are built, Buck needs to go through the DEX merge, alignment, signature and APK packaging for various projects again and continue to the installation procedure. To view the changed effects on the mobile phone, several page switches may be required to enter the modified page. These processes consume a considerable length of time. What's more, Buck does not support Windows and is highly invasive (the whole project needs a major adjustment to be used), which constitutes the threshold for adoption of Buck. But we have to admit that from the perspective of full builds, Buck is second to none. It is also maintained by the powerful Facebook technical teams. Inside Facebook, all the apps use Buck for builds. In China, WeChat app also uses it as a default build solution.

Instant-Run
Last, it comes to Android Studio 2.0 Instant-Run, the official increment solution of Google. First of all, its basic process is similar to that of LayoutCast. Because its code increments are implemented through the hack method at runtime, the app does not need to repeat the original lifecycle after Instant-Run is applied. As a result, if you want to see the effect after you modify the lifecycle methods such as onCreate and onResume, you must restart the process manually. In addition, different mobile phones have different command sets, so there is a chance that the Instant-Run fails. At last, because Instant-Run adopts the hack method, you cannot see the corresponding method stack during debugging packages, which is a huge defect, I regret to say. Finally, like LayoutCast, Instant-Run does not support devices lower than Android 5.0.

Core ideology
Freeline was born integrating the respective advantages of the above solutions in an aim to eliminate their disadvantages. The core technical idea of Freeline stems from Buck and LayoutCast, with further improvements made on the basis to bring the increment ideology into full play.
Its major features are as follows:
Multi-task concurrency, multi-tier caching, minimized increment scope, load on demand, persistent-connection-based install-free dynamic replacement at runtime, baseline alignment triggering mechanism, and debuggable.

Multi-task concurrency
If you have studied Buck, you should be aware that Buck breaks down the tasks on a single pipeline into multiple child task nodes for concurrent execution by project. It streamlines the dependent relations between the nodes and summarizes a directed topology to build various child task nodes through multiple concurrent threads, fully leveraging the multi-core advantage. On a MacBook, it implements 16 threads concurrency by default.
Freeline learns from Buck at its startup. It calculates and prepares the directed topology in advance according to the dependency relationships between projects and tasks for concurrent execution of tasks. By default, it enables eight threads (because of the limited number of projects in Ant Fortune, there is no need to enable too many threads). Next, I will briefly introduce the related information:

Directed topology:
Topology is a kind of graph. Being “directed” ensures the dependency and sequence relations. There can be multiple root nodes,
and a child node can have multiple parent nodes.


Below is a chart that outlines the task order of various projects during the build process in Freeline.


The whole project can be divided into:
Persistent connection between the PC end and the mobile phone end, scanning for file changes in various child projects, building incremental DEX files of various child projects, building incremental resource packages, merging all project DEX files, and transmitting incremental packages.
In the figure above, the forked arrows represent concurrent tasks. Different project may be in different build stages at the same point of time. Freeline will first define the dependency relations between various child projects and their child tasks, as well as the lead tasks and post tasks of every task. Projects at the same layer will be built in concurrency. Eight threads concurrency is adopted by default.

Procedure of a single project:
The overall working mechanism of Freeline has been outlined above. Next we will go into the detail of the tasks in every child project of Freeline: First, we use a figure to illustrate the procedure of tasks in a single project in Freeline builds:


Taking the app project for example, here the app is dependent on the common project and the following steps are involved in the build process:
1. Scanning the file changes in the app project.
2. Based on the scanning result, if both resource and code changes are detected, the inc-code-task and inc-res-task are executed in concurrency.
The task starting with “inc” indicates the task is an incremental task.

About inc-code-task
From top to bottom:
check-r-change:
(Validate whether the R file MD5 changes) If the R file MD5 changes, the new R.java is added to the change list.
begin-code-transaction:
This process backs up the necessary workspace required by the code increments. In case of any errors in one of the following processes, transaction rollback will be triggered for the built products throughout the whole process.
javac:
It gathers the java changes scanned for compilation. If some project that the changes are dependent on (the common project in the previous example) is also being built, the compilation is suspended until the javac build of the lead task is complete, and then the execution will continue.
buck-dx:
Here the previously compiled class file is actually changed to the DEX file. I use “buck-” to depict it because this DX tool is extracted from Buck, and it is faster by around 40% than the native Android DX tool according to tests.
buck-smart-dex:
Similar to above, this tool is extracted from the Buck tool. It aims to reduce the size of the DEX file packaged in the previous step. The finally generated DEX file is the final result of this round of incremental DEX build of this project.

About inc-res-task
begin-res-transaction:
This process backs up the necessary workspace required by the resource increments. In case of any errors in one of the following processes, transaction rollback will be triggered for the built products throughout the whole process.
merge xml:
If the changed file also exists in other child projects, taking the mPaaS architecture as an example, the api, biz, build or tools may have xml files of the same name. Under such circumstances, the corresponding nodes in these xml files should be merged.
merge ids:
If the R file MD5 is discovered to be changed in the above gen-r stage, or ids.xml or public.xml exists in the changed file sets, xml nodes should be merged for the ids.xml and public.xml in the target directory and the ids.xml and public.xml that are newly changed.
gen id files:
This process is the key for achieving resource increments. It generates the
Ids.xml and public.xml in a reversed way using the last built resource package. The two files take part in the compilation for building the incremental resource packages
so that the built resource package keeps consistency with the previous resource package in terms of resource ID. I will detail the principle of this process later on.
build-inc-res:
This process will pass the resource changes sets that we scanned previously into the increment-version AAPT we have adapted as parameters. This tool mainly accomplishes the following tasks:
1. Build the incremental packages. When the final resource package is generated, only the changed resource set after compilation as well as “resources.arsc” and “AndroidManifest.xml” are included.
2. Stay compatible with the mPaaS architecture to solve the Base Package ID issue.
3. According to ids.xml and public.xml, generate ID values in consistency with those in the previous build. If this task has a lead resource task (such as the common in the above example), it will not be executed until the lead incremental resource task build is completed. The final package ends with “.pack”.

Multi-tire caching
For code changes, Freeline introduces caching at the class and DEX layer of various projects. Compiled java files can directly get the data from the Class pool of incremental workspace, and DX-processed class files can get the data directly from the DEX pool. The final result is that every incremental build is a fresh new process and the previous changes will not be included in this incremental compilation process. As a result, the problem in LayoutCast, that is, the solution gets slower with the increase of modified files, does not exist.


For resource changes, Freeline will synchronize the incremental changes in resource files with the corresponding files on the mobile phone end after each incremental package build. The scope of every resource increment build is confined to the set of modified files this time and previous modifications are all synchronized to the mobile phone end in previous sync processes. Similar to code changes, you don't need to build the increments set for previous changes.


Data comparison after multi-tier caching and multi-task concurrency policies are introduced

Build speed (s)
Modification scope (five child projects and 50 changed files)
Minimized increment scope
Freeline will try to minimize the increment scope to a file set that contains only the must-change files in a single modification. It conducts irregular synchronization with the mobile phone end to reduce performance loss because of the increasing modification scope.
In the code layer, Freeline utilizes the multi-tier caching mechanism mentioned earlier. It only compiles the files modified in this round and files previously modified are not within this compilation scope.
In the resource layer, in order to minimize the incremental package size and build costs, we provide an extension tool on the basis of AAPT named IncrementAapt, and make it into three versions for the Linux, Mac OS and Windows platforms for the sake of compatibility across platforms. This tool builds the corresponding incremental package according to the modified resource files and the last resource build results. This incremental package only contains the changed resources and is compressed using the 7-zip tool. The actual size depends on the modification volume, and is usually hundreds of KB. It greatly reduces the time consumption for packaging resources and the final TCP transmission.
Guest