Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Improving the Building Efficiency - Alibaba DevOps Practice Part 17

Improving the Building Efficiency - Alibaba DevOps Practice Part 17

Part 17 of this 27-part series discusses improving building efficiency from two main aspects.

This article is from Alibaba DevOps Practice Guide written by Alibaba Cloud Yunxiao Team

Building is a process of turning source code into deliverables. It includes compilation, but it is not the same as compilation. Even interpreted languages that do not require compilation must build a compressed package or Docker image before deployment. A building process is indispensable in the development and CI/CD stages. The building quality and efficiency have a big impact on continuous delivery. Factors that affect building efficiency include the source code and building dependencies.

Background

Java is the most widely used programming language at Alibaba. Among Java’s building tools, Maven has always been the most important building tool for server applications due to migration costs, ecosystem, and other reasons.

Two main reasons lead to the building performance problems of Maven. First of all, these problems are caused by external factors. From the perspective of applications, some Java applications were created a long time ago and had an increasing number of dependencies. Too many dependencies are accumulated, and there might be risks if they are cleaned up. A typical application has 3,000 JAR packages with dependencies. In addition, a large number of developers from multiple teams are involved in the development of these applications. Therefore, the dependency management cost is high, and dependencies may become complex. Secondly, these problems are partially caused by Maven itself. The tool is not suitable for handling complex dependencies.

The business team needs to establish a dependency management mechanism for the first reason. We re-implemented Maven and launched AMaven for the second reason.

On the other hand, 15% of Alibaba is C/C++ applications. The biggest difference between C/C++ applications and Java applications is that Java applications are built frequently, and it takes a short time each time. C/C++ applications are rarely built but take a long time. For example, some software builds take more than ten hours. In addition to the slow building, the most unbearable pain point for C/C++ developers is that a build can fail in the last few minutes after building for more than ten hours.

Why is the building process of C/C++ applications so slow and easy to fail?

There are two main reasons:

  1. The existing framework cannot guarantee the rigor of compilation and linking of C/C++ applications, resulting in uncertain compilation results and unstable runtime. For example, header files with the same names may have different content at different times or run on different machines during compilation, making the compilation likely to fail.
  2. There are multiple compilation frameworks in the company, including SCons and CMake. This causes a high cost for transferring businesses across teams and troubleshooting compilation problems. From the platform perspective, the real compilation logic cannot be implemented, the performance optimization cannot be unified, and the underlying compiler GCC cannot be upgraded uniformly. In other words, the benefits of new technologies cannot be enjoyed. Therefore, the building process is slow.

For C/C++ applications, we mainly built the upper-layer compilation framework and launched Alimake.

Solution

AMaven

The performance problems brought by Maven will seriously affect the efficiency of continuous delivery. They are mainly reflected from the following points:

  1. IntelliJ IDEA takes a long time to synchronize data. It takes about ten minutes for a typical application.
  2. A single compilation takes a long time. The compilation takes about ten minutes for a typical application.
  3. There are many building steps, which are required in a single delivery process for different environments.
  4. The building process is performed multiple times in the same environment.

The building performance problems also make developers feel less accomplished. They only write a few lines of code, but it takes about the whole morning to refresh the project and compile it locally. At the end of the day, developers spend more time with compilation than writing code.

1
Building Problems Affect the Software Delivery Efficiency

R&D tasks of front-line R&D personnel are far more than performing the operations on the R&D collaboration platform. Before a branch can enter the integration stage, most of the R&D tasks are done locally and offline. Therefore, it is also important to improve the efficiency of finishing tasks locally and offline. Starting from the new Java building tool AMaven, we have also extended our perspective of efficiency improvement from the online collaborative R&D platform to offline R&D tasks.

We implemented a new tool called AMaven based on the Maven protocol and by following the caching and increment principles. When AMaven is being used, Apache Maven is also adopted in the background for building, and the compilation results of the two tools are compared to ensure the accuracy of the building result. The whole process is Maven-based and transparent to users to achieve no cost and risk.

2
No Cost and Risk

3
Compare the Results to Ensure No Risk

AMaven solves the complex dependency problems by establishing, caching, and sharing a dependency tree. Whether the cache of a dependency tree is valid depends on the pom file in the source code and the changes of the dependent snapshot. When the snapshot is changed, the cache of the dependency tree becomes invalid, and the dependency tree needs to be regenerated. AMaven also optimizes the dependency tree generation process to improve the efficiency of dependency tree generation using the dependency traversal algorithm and opens the traversal depth parameters for users to fine tune.

The dependency tree information will also be saved as important metadata of products, which will be detailed in the article about the product metadata.

4
Dependency Caching and Sharing

In addition to using cache and increment, AMaven is Client/Server-based. In other words, some computing capabilities are moved to the server. The building process of AMaven is shown below:

5
A Typical AMaven Building Process

AMaven also provides the capabilities of detecting circular dependencies and dynamically executing plug-ins. This cannot directly improve the building speed, but it can speed up the troubleshooting of building problems caused by dependencies.

Let's take a look at the benefits AMaven brings to users. The online CI/CD platform data shows that AMaven builds Java applications in seconds, and 44% of building tasks of Alibaba's Java applications are completed within 30 seconds. In addition, the building efficiency of typical applications with large dependencies can be improved by ten times.

From the Offline Local Development Data: AMaven can reduce the building time by 50% whether it is used in the command line or the IntelliJ IDEA. In IntelliJ IDEA, the building task can be completed within ten seconds after refreshing the project and switching branches.

AliMake

AliMake mainly focuses on the following two points:

  1. Specification: First, create a new strict configuration file named target. All input compilation parameters must strictly follow clear definitions. The file will be translated into a makefile. This can help cultivate a culture of rigorism and self-discipline for engineers to some extent. Then, AliMake creates a new repository named alicpp, which stores the dependencies originally compiled on the machine, ensuring the irrelevance of the compilation environment and strong consistency of the compiling results.
  2. Efficiency: AliMake has integrated industry-leading technologies, ccache and distcc, and exclusively developed technologies, such as distributed linking. It unifies interfaces, enables the large-scale application of expert experience, fine tunes the compilation parameters, and institutionalizes the GCC upgrades so we can keep up with the progress of the compiler.

The architectural design of AliMake is similar to AMaven. Some of the computing capabilities are moved to the server side. This way, the resource consumption of the client can be reduced, and the performance bottleneck caused by hardware and configuration on the client can be solved. In addition, the server and the client can share resources, such as dependency cache and intermediate product cache.

6
AliMake Architecture

AliMake is used by multiple Alibaba products, including DingTalk, Alibaba Cloud storage products, OSS, Apsara Distributed File System, Job Scheduler, and Ant Financial AI. The average building efficiency is increased by 30%, and the efficiency can be increased by 70% under optimal circumstances.

Summary

We made improvements from two aspects to improve the building efficiency: space (product size) and time (using tools to speed up the building). The native data generated during the building process (such as dependencies) is used to empower continuous delivery and safe production, enabling an efficient and reliable release.

java c++ DevOps continuous delivery Maven CI/CD Business Efficiency Compiler C Language
0 0 0
Share on

Read previous post:

Application and Change-Based Delivery Model - Alibaba DevOps Practice Part 16

Read next post:

Enhancing Delivery Efficiency Based on Product Metadata - Alibaba DevOps Practice Part 18

Alibaba Cloud Community

949 posts | 221 followers

You may also like

Comments

Alibaba Cloud Community

949 posts | 221 followers

Related Products

More Posts by Alibaba Cloud Community

See All