This article is from Alibaba DevOps Practice Guide written by Alibaba Cloud Yunxiao Team
Building is a process of turning source code into deliverables. It includes compilation, but it is not the same as compilation. Even interpreted languages that do not require compilation must build a compressed package or Docker image before deployment. A building process is indispensable in the development and CI/CD stages. The building quality and efficiency have a big impact on continuous delivery. Factors that affect building efficiency include the source code and building dependencies.
Java is the most widely used programming language at Alibaba. Among Java’s building tools, Maven has always been the most important building tool for server applications due to migration costs, ecosystem, and other reasons.
Two main reasons lead to the building performance problems of Maven. First of all, these problems are caused by external factors. From the perspective of applications, some Java applications were created a long time ago and had an increasing number of dependencies. Too many dependencies are accumulated, and there might be risks if they are cleaned up. A typical application has 3,000 JAR packages with dependencies. In addition, a large number of developers from multiple teams are involved in the development of these applications. Therefore, the dependency management cost is high, and dependencies may become complex. Secondly, these problems are partially caused by Maven itself. The tool is not suitable for handling complex dependencies.
The business team needs to establish a dependency management mechanism for the first reason. We re-implemented Maven and launched AMaven for the second reason.
On the other hand, 15% of Alibaba is C/C++ applications. The biggest difference between C/C++ applications and Java applications is that Java applications are built frequently, and it takes a short time each time. C/C++ applications are rarely built but take a long time. For example, some software builds take more than ten hours. In addition to the slow building, the most unbearable pain point for C/C++ developers is that a build can fail in the last few minutes after building for more than ten hours.
Why is the building process of C/C++ applications so slow and easy to fail?
There are two main reasons:
For C/C++ applications, we mainly built the upper-layer compilation framework and launched Alimake.
The performance problems brought by Maven will seriously affect the efficiency of continuous delivery. They are mainly reflected from the following points:
The building performance problems also make developers feel less accomplished. They only write a few lines of code, but it takes about the whole morning to refresh the project and compile it locally. At the end of the day, developers spend more time with compilation than writing code.
Building Problems Affect the Software Delivery Efficiency
R&D tasks of front-line R&D personnel are far more than performing the operations on the R&D collaboration platform. Before a branch can enter the integration stage, most of the R&D tasks are done locally and offline. Therefore, it is also important to improve the efficiency of finishing tasks locally and offline. Starting from the new Java building tool AMaven, we have also extended our perspective of efficiency improvement from the online collaborative R&D platform to offline R&D tasks.
We implemented a new tool called AMaven based on the Maven protocol and by following the caching and increment principles. When AMaven is being used, Apache Maven is also adopted in the background for building, and the compilation results of the two tools are compared to ensure the accuracy of the building result. The whole process is Maven-based and transparent to users to achieve no cost and risk.
No Cost and Risk
Compare the Results to Ensure No Risk
AMaven solves the complex dependency problems by establishing, caching, and sharing a dependency tree. Whether the cache of a dependency tree is valid depends on the pom file in the source code and the changes of the dependent snapshot. When the snapshot is changed, the cache of the dependency tree becomes invalid, and the dependency tree needs to be regenerated. AMaven also optimizes the dependency tree generation process to improve the efficiency of dependency tree generation using the dependency traversal algorithm and opens the traversal depth parameters for users to fine tune.
The dependency tree information will also be saved as important metadata of products, which will be detailed in the article about the product metadata.
Dependency Caching and Sharing
In addition to using cache and increment, AMaven is Client/Server-based. In other words, some computing capabilities are moved to the server. The building process of AMaven is shown below:
A Typical AMaven Building Process
AMaven also provides the capabilities of detecting circular dependencies and dynamically executing plug-ins. This cannot directly improve the building speed, but it can speed up the troubleshooting of building problems caused by dependencies.
Let's take a look at the benefits AMaven brings to users. The online CI/CD platform data shows that AMaven builds Java applications in seconds, and 44% of building tasks of Alibaba's Java applications are completed within 30 seconds. In addition, the building efficiency of typical applications with large dependencies can be improved by ten times.
From the Offline Local Development Data: AMaven can reduce the building time by 50% whether it is used in the command line or the IntelliJ IDEA. In IntelliJ IDEA, the building task can be completed within ten seconds after refreshing the project and switching branches.
AliMake mainly focuses on the following two points:
The architectural design of AliMake is similar to AMaven. Some of the computing capabilities are moved to the server side. This way, the resource consumption of the client can be reduced, and the performance bottleneck caused by hardware and configuration on the client can be solved. In addition, the server and the client can share resources, such as dependency cache and intermediate product cache.
AliMake Architecture
AliMake is used by multiple Alibaba products, including DingTalk, Alibaba Cloud storage products, OSS, Apsara Distributed File System, Job Scheduler, and Ant Financial AI. The average building efficiency is increased by 30%, and the efficiency can be increased by 70% under optimal circumstances.
We made improvements from two aspects to improve the building efficiency: space (product size) and time (using tools to speed up the building). The native data generated during the building process (such as dependencies) is used to empower continuous delivery and safe production, enabling an efficient and reliable release.
Application and Change-Based Delivery Model - Alibaba DevOps Practice Part 16
Enhancing Delivery Efficiency Based on Product Metadata - Alibaba DevOps Practice Part 18
1,011 posts | 247 followers
FollowAlibaba Cloud Community - February 4, 2022
Alibaba Cloud Community - March 8, 2022
Alibaba Cloud Community - March 2, 2022
Alibaba Cloud Community - February 18, 2022
Alibaba Cloud Community - February 14, 2022
Alibaba Cloud Community - February 5, 2022
1,011 posts | 247 followers
FollowAn enterprise-level continuous delivery tool.
Learn MoreAccelerate software development and delivery by integrating DevOps with the cloud
Learn MoreReach global users more accurately and efficiently via IM Channel
Learn MoreAccelerate static and dynamic web content in a fast, reliable, and safe way using Secure DCDN (Dynamic Route for CDN)
Learn MoreMore Posts by Alibaba Cloud Community