Gordon
Assistant Engineer
Assistant Engineer
  • UID622
  • Fans1
  • Follows0
  • Posts52
Reads:1041Replies:0

Freeline - second-level compilation solution on the Android platform (2)

Created#
More Posted time:Dec 7, 2016 9:40 AM
Load on demand
Freeline defers task execution until it is truly needed. Take the javac compilation of the R file for example. If only the resource file is modified, even if a new resource file is added, such as a new ID, a new image or a new layout, and the operation leads to inconsistency between the ID sets of the new R file and the old R file, as long as the java file is not modified at this time, the compilation of the R file will not be triggered. That is to say, if you only modify the resource without modifying the java code, no matter whether the application's ID set has been actually changed, Freeline will build the incremental resource package at an extremely low cost and push the package to the mobile phone. As a result, the interface will be refreshed directly in the current activity without the need to restart the process. For the compilation of a new R file, Freeline will defer the task execution until when there is a change to the java file in the project. This approach also ensures the readiness of a corresponding value when the code truly needs the newly added ID value of the R file. Before the code is changed, the process does not need to be restarted, speeding up the refreshing efficiency.

Debuggable
Android Studio Instant-Run adopts the hack method solution, so modified methods may not be debuggable. Incremental classes built by LayoutCast may also fail to display the parameter values in the debug mode. Freeline makes some processing in this aspect and ensures the consistency between the incremental build class file and full build class file without impacting routine debugging.

Persistent-connection-based install-free dynamic replacement
Install-free dynamic replacement of Freeline is consistent with that in LayoutCast and Android Studio 2.0 Instant-Run, and install-free dynamic replacement is also the biggest advantage of the two incremental build solutions. No re-installation of apps is required throughout the build process and code and resources are replaced dynamically, saving the process of app installation and process restart to enter corresponding interfaces. The entire interaction procedure is shown in the figure below:

1. A TCP socket will be established on the mobile phone end as the server.
2. The PC end will set up a socket connection with the mobile phone end.
3. The PC end and the mobile phone end will conduct interaction through the self-defined protocol. The PC end will query the mobile phone status, such as getting the baseline package version and SDK version number of the mobile phone end, checking whether the current mobile phone supports incremental resources, and obtaining the current activity name. Later it will transmit the incremental package, and the mobile phone end will return the incremental build result to the PC end, and so on. The whole communication process will be carried out in the same persistent connection.
4. After the incremental package is completely synchronized, the mobile phone end will decide the next operation based on whether the current change is a code change or a res change. If it is the latter case, it will directly restart the activities in the entire activity stack; if a code change is involved, it directly restarts the current process. Because of the management on the activity stack in the Android system, if a process is killed but some activities are still alive in the activity stack, these activities will be re-created following the original stack sequence when this app is restarted. The final result will be that after the restart, the interface will show the last-displayed activity (there is a special case here that if singleTask or singeInstance has been set in the launch mode of this activity, all the other activities in the stack except the last activity will be cleared after the restart. This involves the management mechanism of Android on activities which I will not detail here. If you are interested, you can Google it.) If you press the Back key, the UI will also display the activities following the original activity sequence in the stack.

Baseline alignment triggering mechanism
Freeline will re-build the baseline package under the following circumstances:
1. If you run the git pull command, or you modify a large number of files at a time, the incremental package size will soar, impairing the subsequent transmission and dexopt speed on the incremental package after the mobile phone is restarted. Considering that this is a rare case, there is no need to influence the subsequent incremental build speed for one change.
2. Modification cannot be implemented by relying on the increments: Modify AndroidManifest.xml, change the third-party JAR reference,
and depend on the compile-time aspect, annotation or other functions implemented by code pre-processing plug-ins.
3. The debugging mobile phone is changed, or the same debug mobile phone has installed an installation package inconsistent with that of the development environment.
Since there may have been multiple incremental builds before the baseline package is re-built, you should conduct a full build for the corresponding modules of these incremental builds when re-building the baseline package, so as to ensure the latest baseline package contains all history modifications. The entire procedure is shown in the figure below: (A, B and C are three different child projects respectively)

The head acts as the pointer and points to the latest baseline package status. The base is the initial baseline status. After three incremental builds, the app status on the mobile phone end will change to the result of base + three incremental builds. In the above example, the three incremental builds involve three modules of A, B and C. As a result, during the process of triggering the baseline package alignment, A, B and C will be built following the original full build method. Like the incremental package build, the sequence of the full package build will observe the dependency relationships of A, B and C in order. The projects of the same level will be built concurrently. After the build is completed, the package will be re-installed on the mobile phone. Afterwards, the app on the mobile phone end will include the changes of A, B and C in the full package mode, and the previous three incremental packages will be cleared at the first startup after the overwrite installation. At this time, the baseline pointer head will point to the latest base (with new A, B and C) from the initial base. By now, the whole baseline alignment is completed. If any exception occurs during the process, a baseline alignment process will be performed at the next run to ensure that the mobile phone end has the latest full package installed.
Baseline alignment verification mechanism
In the section above, we introduced the overall train of thought for baseline alignment. Next we will illustrate the key idea of baseline alignment verification:

1. During the full package build, package the current timestamp in the assets directory. This value is used to ensure the consistency of the full package.
2. After each incremental package transmission, the mobile phone end and the PC end jointly maintain a self-increasing sync ID. Upon the completion of each successful transmission, this ID will trigger updates. This value is used to ensure the correspondence between the development status in the development environment and the incremental development package status on the mobile phone end.
3. Before each incremental package transmission, the mobile phone end and the PC end will generate a verification code based on the above two values and collate this verification code. If the verification codes on the two ends are inconsistent, the collation is deemed as failed and baseline alignment is needed.

Process-level exception isolation:
The socket TCP server of Freeline is run in a separate process. The reason for process isolation is to prevent incremental transmission failures when the developed incremental part is transmitted to the main process and causes a crash. So the TCP transmission is separated to an independent process to ensure the continuity and stability of the transmission process. Actually this is also in line with the “separation of lighweight and heavyweight tasks” principle, by leaving the heavyweight part of refreshing and replacement that easily causes the main thread to crash, and leaving the stable part of connection establishment, transmission and baseline alignment to separate processes.

Increment principle
Code increment:
The code increment is the same with the mainstream hotpatch solution in the industry by embedding DEX to the system DexList. There have been many introductions about its principles on the internet. Here I will skate over it:
The system will try to find the class in the BaseDexClassLoader finally.
Then it calls the DexPathList.

In it, the DexFile corresponds to the class.dex, class2.dex and so on in the default installation package. After Google started to support MultiDex, the build tool by default subpackages according to the 65536 method and the memory limit of LinearAlloc. Usually a large app will have multiple dex files. From the code above, we can see that the class searching starts from the beginning of the dex array. If the corresponding class is found, the search will stop, which offers us an opportunity to utilize this feature for implementing increments.
When the app is started, the incremental dex file we have prepared can be injected to the very beginning of DexElements through reflection, and the entire incremental deployment is completed.

Resource increment:
Resource increment is the most time-consuming part during the Freeline development, and also a feature that distinguishes Freeline from other build methods. We have mentioned earlier that after a resource is changed, LayoutCast and Instant-Run actually re-package the full data of res resources and push it to the mobile phone to replace the entire resource package. Therefore, the bigger the number of resources and the larger the size, the longer the build will take.
Let's first talk about what problems should be solved for developing a resource increment feature:
1. How can the incremental package resource ID be compatible with the baseline package resource ID?
2. How to efficiently build a resource package that only contains the changed sets?
3. How to make the incremental package built above take effect on the mobile phone end?

With these questions, let's proceed with the introduction step by step:
The first question goes first: 1. How to ensure the consistency between the baseline package resource ID and the incremental package resource ID?
1. Regarding the forward compatibility of resource package ID, the industry generally solves this issue by making the public.xml and ids.xml generated by the last resource package join the subsequent resource compilation. There are mainly two solutions to generate the above two files, as found below:
Generate the files during app runtime through reflection of the R class field
This solution has been mentioned earlier. It has a fatal defect and the reflection process involves tens of thousands of fields, being very inefficient.
Decompile the resource package using ApkTool
This solution requires a reverse export of all the resources which are unzipped from the resource package and decompiled back to the original file to generate the corresponding ids files. With the growing number of resources and increasing sizes, the processing duration also increases.

The idea of Freeline for this problem is to utilize the R.java for the last res compilation to export the two files needed for reserving the ID. This function is extracted to a separate tool “id-gen-tool”. This tool will filter out the enumeration constants based on the context features of IDs generated by the enumeration constants to solve the memory out-of-bounds exceptions caused thereof.

Since the whole process requires analysis and export of only one file - R.java, without the need to unzip the APK and decompile resources in the APK resource package, the entire process is basically not influenced by the number and size of the resources in the resource package. In addition, the process is run on the PC end, so it is faster by more than 90% than on the mobile phone end. Below is the data comparison:


With 30 MB of resources, the id-gen-tool is 90% faster than the app reflection solution, and more than 95% faster than the ApkTool decompilation solution. With the increasing amount of resources, the gap gets even prominent.

Detailed issues of id-gen-tool
With these two files, the resource ID issue is settled. But is it really this simple? Wait a second. Place these two files under the values directory in the resource directory, compile the resource, and an expected issue emerges:
Let's look at the definition in the styles.xml file.
< style name="Animations.Pop">
        < item name="@android:windowEnterAnimation">@anim/pump_bottom < /item>
        < item name="@android:windowExitAnimation">@anim/disappear < /item>
    < /style>

What is the ID corresponding to the generated R.java:
public static final class style {
        public static final int Animations_Pop=0x1f0b002c;
        ………………….    
    }

That is to say, the resource name in the R.java does not include “.” at all, and the ID generated through R.java will be like this:
< resources>
    < public type="style" name="Animations_Pop" id="0x1f0b002c" />
    ………………………
< /resources>

Consequently, the final result is that the “Animations_Pop” resource cannot be found during resource compilation, leading to compilation errors. What's more, because we cannot deduce whether it is “.” or “-” in the original resource definition based on the R.java variable name, it is not feasible to generate the ID file in a reverse direction through R.java. But fortunately the AAPT program is also with us. As long as we make AAPT compatible with this situation, the above solution will work. At last, we can expand the resource searching policy of AAPT, namely when the resource cannot be found, we will try to replace the “-” in the resource name to “.” to continue the search. In this way, the above issue is also solved.


On a final note, after the baseline package ID is fixed, the newly added resource will not influence the access to the original resource ID. That is to say, on this premise, we don't need to ensure the consistency between the number of resources in the incremental package and that in the baseline package. This also solves the possible asymmetry with the baseline package ID arising from the introduction of new resources during daily development.

2. Next we will introduce the second point: How to efficiently build a resource package that only contains the changed sets?
The process figure is as follows:


We marked the optimizable technical points throughout the resource package build process in red circles in the figure above: In fact, after the resource ids.xml and public.xml files are generated in the previous step and placed into the values directory for compilation, even if you don't compile the unchanged layout resources and AndroidManifest.xml, there is no impact on the final generated resoucres.arsc. That is to say, with the resource ID retained, you only need to compile the changed xml files to update resoucres.arsc.
In the previous scanning, we have known all the resources files with changes. Python will capture the relative paths of these resource files and pass the information into the incrementAapt tool as the ‘—buildIncrement’ parameter. In the resource compilation process, if the resource is not changed, we utilize the compiled resource in the last resource package as the cache. We can let the unchanged file read directly from the compiled resources and the whole process does not require re-compilation on the unchanged resources. (Because of the many changes to the code in this part, I will not paste it here. It will go open-source after it is well organized.)

During the final package into the APK: we also modified the file packaging process. The incrementAapt only packages the compiled resources corresponding to the modified files:

After the whole process is completed, the final build package only contains the changed resource sets as well as “resoucres.arsc” and “AndroidManifest.xml”.
Here I want to explain: why we need to package “resoucres.arsc” and “AndroidManifest.xml”?
Because when there are newly added resources, the “resoucres.arsc” will change and the reference of the newly added resources in the code is implemented through updating “resoucres.arsc”. Here we package the “AndroidManifest.xml” because after SDK 19, the underlying AssetsManager > addpath process will trigger the validation of the res resource package. Resource packages without the “AndroidManifest.xml” will be deemed as illegal and won't be successfully added.
Guest