Assistant Engineer
Assistant Engineer
  • UID622
  • Fans3
  • Follows0
  • Posts52

Freeline - second-level compilation solution on the Android platform (3)

More Posted time:Dec 7, 2016 10:02 AM
Optimization data for various resource compilation steps
Taking 30 MB of resources for example, the data before and after the overall resource compilation process optimization is shown below:

Next I will briefly introduce the various steps. The detailed procedures can be found in Lao Luo's Blog I mentioned earlier.
slurp up res: collect the resources.
makeFileResources all resource: add the collected resources in the previous step to the memory. If there are images, the image resources will be
processed in this step.
compile value: compile the resources under the “value” directory.
makeFileResources for color and menu: compile the resources in the “color” and “menu” directories.
generate all bag attr: allocate resource ID for the “bag” type.
compile all xml: This step is where the real compilation and flattening on the resources contained in the layout, anim, animator, interpolator, transition, xml, color and menu directories are performed.
flatten gen resources.arsc: generate the “resources.arsc” file according to the information collected above.
gen r file: Generate the R.java.
APK Bundling: package all the compiled resources.

We can see from the data comparison that: The main speed improvements occur to the compile all xml (compiling xml files), APK Bundling (packaging APK), and makeFileResources all resource (pre-processing image resources) steps. The traditional approach of AAPT compilation and packaging may take nearly 3.5 seconds, but the entire build in the above method only takes 300 milliseconds. This is 90% faster than the former method. With the increasing number of resources and the size of the entire package, this gap will grow wider and wider.

The third point: How to make the incremental package built above take effect on the mobile phone end?
Through in-depth analysis on the underlying layer of AssetsManager, we found that res actually supports the form of directory. So the idea about how the entire incremental package takes effect is presented and the procedure is as follows:
1. In the first running of incremental build, the mobile phone end will unzip the base.apk (here it is the resource JAR path corresponding to the bundle in mPaaS) of the baseline package:

2. After this, the resDir directory will contain all the files in this resource package. After this, we can
point the path corresponding to AssetsManager to resDir. In this way, the resources that the UI actually corresponds to will be from the resDir directory. According to the introduction earlier, after the resource change, Freeline will trigger a synchronization after it passes the incremental resource package inc.pack into the mobile phone through the TCP connection to synchronize the changes to the incremental package with the mobile phone end. The procedure is shown below:

The mobile phone end will unzip the inc.pack and then write the buffer after the unzipping directly to the relative location in the resDir directory. The whole process only involves two steps, namely extracting from the zip package and writing to the file system. In the last step, the only task remained will be clearing the corresponding cache of the resources, re-establishing the new resources and enabling the app to use it. There have been many introductions on the internet already so I will not repeat it again here. To make the mPaaS architecture effective, it is nothing but to find the resources where the corresponding bundle is located, clear its cache and refresh the UI again.

Data comparison of resource incremental build:

With 40 MB of resources, it takes only 600 milliseconds to complete a resource incremental build with the above solution, while it may take more than 4 seconds with other solutions. The incremental package size from this solution is only 300 KB, while that from other solutions is 5 MB.
Here I want to mention one point: there is another solution to make the incremental resource package take effect, that is, to adopt the “overlay” solution of the system.
That is, you can set the index items corresponding to the resource ID where “layout”, “drawable”, “color”, “anim”, “xml”, “raw”, “animator”, “interpolator”, and “menu” directory types are located during “resources.arsc” build to “NO_ENTRY”, and place the incremental resource package path during the generation of AssetsManager at runtime in order before adding the full package, so that you can utilize the system's search mechanism to overwrite the changed resources.
One of the reasons why Freeline didn't select the above solution is that:
1. Because of the designed limit of the system overlay, the resource cannot realize the “newly add” feature, but only supports modification. This is very inconvenient in daily development.
2. You must make sure to involve all the changed resource files since the full resource package build in the compilation and package in every incremental resource package build. That is to say, with the increasing modified resource volume, the number of resources for compilation and packaging will grow bigger and bigger as time goes by. While the above solution, simply put, is capable of staying completely out of the influence of accumulative changes. After each modification is synchronized with the mobile phone, the modification is regarded as cleared and subsequent compilation does not need to pull the previous modified resource files for compilation and packaging. This also enables Freeline to maintain a stable performance during UI modification without the influence of the accumulation of modification scopes.

Resource index cache
The resource.arsc is the index file storing the Android resource ID indexes. In some large apps, the arsc file may have a big size, ranging from 6 MB to 10 MB in general. Freeline adopts an optimization strategy before arsc packaging: when the resource modification does not involve the arsc updating, it will not package arsc into the incremental package to avoid useless packaging and TCP transmission. The policy that Freeline adopts is to pass the MD5 of the previous arsc as the incoming parameter, and calculate the MD5 of the arsc memory block at the C++ layer in advance during packaging of the AAPT compilation process. The packaging will be triggered only when any inconsistency is found with the MD5 of the previous arsc. In large apps (with a large resource volume), the optimization of the TCP transmission + packaging + unpacking can shorten the processing time by 3 to 5 seconds.

Behind full-platform coverage
1. The build process selects the Python + Java as the build languages. In specific, the Python is responsible for scanning and scheduling various
tools (dx, smart-dex, merger, increment-res-tool), setting up TCP connections with the mobile phone end and transmitting the incremental packages among others.
2. The C++-compiled IncrementAapt is compiled into runtime libraries for three different platforms to achieve cross-platform compatibility.

3. On non-Art phones, the code compatibility scheme is to dynamically modify the Class bytecode of the baseline package using Asm technology during the compilation stage, and insert the external DEX reference in every class constructor so that the security check on the class by the DVM can be avoided. We call this “hackbyte”.
Here I will introduce its principle:
The dexopt step in the installation process with the installation package will scan all the Class files in the DEX file. When all the classes directly referenced in the class files are in the same DEX of this Class, this class will be labeled with “CLASS_ISPREVERIFIED”.

The classes labeled with “CLASS_ISPREVERIFIED” will validate the corresponding direct reference classes in the DVM memory during DVM's loading of the Class at runtime. If a certain class is not in the same DEX with the direct reference class, the error of “pre-verification” is reported directly and this class is unable to be loaded (Note: the class labeled “CLASS_ISPREVERIFIED” cannot be loaded, instead of its direct reference class). That is to say, if the class we push through the incremental package acts as the direct reference class of other classes, these classes that reference the classes in the incremental package may suffer verification failures during loading.

In fact, the above step is also a security solution of Google to prevent external DEX injection, that is, to ensure the consistency between the DEX location relation of the class and its direct reference class at runtime and that at the installation.
Through the above analysis, as long as the direct reference class of a class and this class are not in the same DEX, we can avoid labeling this class “CLASS_ISPREVERIFIED”. Next, you only need to dynamically modify the Class bytecode through ASM technology after compiling your own project code, and embed a class from another DEX for all the classes of your project. Pay attention here that we only need to conduct injection for the project we can modify, and this step is not required for third-party JAR packages. Because of the dependency relationship, the third-party JAR package will not reference our project code in turn, so the above issue does not exist.
At last, after we embed the code, the de-compiled code is like this:

The reason for choosing the constructor embedding is because it does not increase the number of methods. In specific, the ClassVerifier.class is from a separate DEX. This DEX only has one class - the ClassVerifier.class. When the app is started, this DEX is injected to the beginning of the above-mentioned DexList. In V5.x or lower versions, this scheme will take effect.
One last note, in the field, this solution is also applied to hotpatch. The hotpatch for QQ Zone in domestic mobile phones is implemented in this way. We apply it to the incremental build solution.
From research, in Art, the dexopt process will optimize the basic types in the final class. All the access to static variables of the final class will be optimized to the access through the offset method. For example:
final class A {
    public static int a = 1;
    public static int b = 2;

Supposing in our development process, we insert a new variable before a: public static int a0 = 0; If you perform code increments with the above solution, the access to a from other classes may get the value for a0, and get the value for a for access to b. That is, all the access to values of the static
int type has been moved downward by one place, leading to wrong values returned to other classes from this patched class. The most typical case is the R.java file. If Buck or a similar tool is used to build the full package, the field in the generated R.java file is not final. When adding a new resource, even if we have solved the resource ID consistency through the solution we mentioned earlier, we cannot ensure that the newly added resource ID can be accessed correctly by other classes, and may even lead to garbled values returned to other classes which access the resource ID in the “R.xxx.xxx” form.
On this point, Freeline also makes some processing. The current solution is to disable android:vmSafeMode in Manifest.xml when developing the environment.
Due to the limited length, I plan to introduce the optimization and solution principle in Art in a separate article and will not detail it here.

Data comparison

We can see that: compared with the traditional Maven build, Freeline can increase the performance by dozens of times in the incremental mode. Compared with the several mainstream build methods in the industry, Freeline still outperforms them by several times in terms of speed.


We can see that compared with LayoutCast and AS2.0 (Android5.0 or lower versions are not supported on the mobile phone end), Buck (Windows is not supported on the PC end) and other build methods, Freeline boasts a wider platform coverage.