EMAS Mobile DevOps solution-Mobile DevOps-Alibaba Cloud Developer Community

ali so native applications R & D platform EMAS Peng Zhao (state Shepherd)

1. What is mobile DevOps

1) the well-known DevOps

at the time point of 2020, DevOps is no longer a new concept. I believe everyone has their own understanding more or less. However, when we need to describe exactly what DevOps is, it seems difficult to explain clearly. In fact, there is still no definition in the industry that can be recognized by everyone. The reason why it is difficult to be defined accurately is that DevOps is actually a collection of ideas or even a set of ideas, it is difficult to be visualized. The word "DevOps" can be literally understood as the whole life cycle of software from Dev(Development, Development) to Ops(Operations, operation), but what exactly is the definition of DevOps? Among many definitions of DevOps, I personally think the definition of Azure DevOps [1] is more accurate and specific:

DevOps is a compound word for development (Dev) and Operation (Ops). It combines people, processes, and technologies to continuously provide value to customers. What does DevOps mean to the team? DevOps enables isolated roles (development, IT operations, quality engineering, and security) to coordinate and collaborate to produce better and more reliable products.

By adopting the DevOps culture, practices, and tools, the team can better respond to customer needs, enhance confidence in the built applications, and achieve business goals faster.

There are several key information in this definition to summarize: ① the combination of people, processes and technologies ② DevOps allows isolated roles to coordinate and collaborate ③ DevOps is a concept, we should not only establish culture, but also have the support of automation tools. ④ The purpose is to produce better and more reliable products faster.

2) from DevOps to mobile DevOps

what we often discuss about DevOps is actually server-side DevOps. Since DevOps is an excellent software delivery concept, why not apply DevOps to mobile delivery? This is the mobile DevOps we are going to introduce today. Mobile DevOps differs greatly from server DevOps due to the differences between mobile and server scenarios. It is mainly reflected in the following aspects:

automated construction of mobile applications is more complex

• Build a fragmented environment

the Android and iOS platforms need to build a building environment based on different operating systems and build toolchains. Even if the same platform builds a toolchain, version fragmentation exists. For example, Android build dependent Android SDK, gradle must be supported by multiple versions at the same time. Xcode and Ruby versions that iOS depends on must be supported by multiple versions at the same time.

• Building a mobile client involves data security issues such as certificate hosting.

• The Mac devices that iOS depends on are non-standard devices in the data center.

Mac devices do not belong to standard servers and cannot be deployed in standard data centers. You usually need to create a self-built Mac Data Center, which is also a challenge for O & M and stability.

Automatic Building is an essential feature in DevOps, which requires mobile DevOps to solve the problems of automatic client building and one-click packet delivery through technical means.

Mobile terminals are heavily fragmented, and application delivery compatibility is a huge challenge.

Different from the consistency of the server deployment environment, the mobile application runtime environment is very fragmented, and the compatibility test coverage is much more difficult than that of the server. The fragmentation of mobile terminals is especially serious in Android system, which is mainly reflected in the following aspects:

• Fragmentation of mobile phone models

Android there are many mobile phone manufacturers and numerous models in the market, different manufacturers will "optimize" the system at the underlying level. Theoretically, any model test that cannot be covered may face compatibility problems, the following figure shows 2020. According to the latest distribution of Android Top models of Baidu Statistics and Traffic Research Institute [2] In October, the market occupancy rate of the Top 10 models is less than 15%, which shows the serious fragmentation of models.

• Operating system version fragmentation

differences in operating systems have a more direct impact on application running. It is common that major version upgrades lead to application incompatibility. Each release of a major version of an operating system is a test of application compatibility; users of the old system cannot be abandoned while considering compatibility with the new system. The following figure shows 2020. According to the latest Android version distribution data of Baidu Traffic Research Institute in October, 10.0 of the Android have been released for more than a year, and the market occupancy rate is less than 50%. The operating system before 2 still occupied the mainstream.

Due to the fragmentation of terminal devices, mobile DevOps is required to have mobile testing capabilities and complete a large number of real-machine compatibility tests automatically.

Long app release and update cycle

A new version of the application may be released within two weeks, and the update rate will not exceed 50%. Unlike the server, the software of all servers can be released in a very short time. A long release cycle means a higher cost of making mistakes. It may take a long time for a Bug version to be released through updates and upgrades.

This requires mobile DevOps to have a complete phased release mechanism to avoid publishing problematic applications to users at one time. On the other hand, once a Bug version has been released, mobile DevOps is required to have the hotfix capability. You can release incremental patch packages to fix bugs more easily and quickly.

Mobile applications run on massive mobile devices

unlike server services running in a specific cluster, they can be managed and maintained in a unified manner. Mobile applications run in a mobile phone, and for Super apps like mobile Taobao, it is a massive device of hundreds of millions.

This requires mobile monitoring products to implement mobile O & M monitoring through big data technology, and even requires the remote log function to pull error logs from specified devices to locate and troubleshoot errors.

Based on the above points and referring to DevOps's definition of software delivery lifecycle, the following table summarizes the mobile DevOps application lifecycle and capability requirements for each phase:

2. What is Mobile DevOps

1)Mobile DevOps is a concrete implementation of EMAS Mobile DevOps concept.

First, let's introduce EMAS(Enterprise Mobile Application Studio), which is a leading domestic cloud-native Application research and development platform (Mobile App, H5 Application, Mini Program, Web Application, etc.) from Alibaba Cloud, based on a wide range of cloud-native Technologies (Backend as a Service, Serverless, DevOps, and low code), it is committed to providing enterprises and developers with one-stop application R & D management services, covering development, testing, O & M, and, the entire lifecycle of applications such as operations. For more information about EMAS, see The EMAS details page on the Alibaba Cloud official website. Mobile DevOps is a concrete product output of the Mobile DevOps concept of EMAS. It is a mid-axis product of EMAS. It works with all EMAS products to realize the Mobile DevOps concept mentioned above. Mobile DevOps implements the linkage and complete closed loop of EMAS products isolated in each lifecycle of applications as shown in the preceding figure, and upgrades EMAS from the Mobile middleware platform to the Mobile R & D platform. Mobile DevOps combined with the following EMAS products to form the Mobile DevOps of EMAS: R & D domain: Mobile DevOps test domain: Mobile test release domain: Mobile DevOps O & M domain: Mobile monitoring, Mobile hot repair operation domain: Mobile push, mobile user feedback

2) The History of Mobile DevOps

Mobile DevOps is a commercial output version of the Mobile R & D platform within the group. It was first developed by Alibaba Cloud and Mobile Taobao in 2017, the first public cloud version was released in April 2020. The following figure shows the development history of Mobile DevOps. It can be said that the development history of Mobile DevOps is actually the development history of Alibaba Group's Mobile R & D technology, which is the accumulation of Alibaba's Mobile technology and engineering R & D philosophy in the past ten years.

3) Current situation of Mobile DevOps

apsara stack has begun to take shape Mobile DevOps, Apsara stack is mainly designed for key customers, especially those who are undergoing digital transformation. These customers have high security requirements and can only accept the mode of Apsara stack deployment, at the same time, they are willing to invest in improving the R & D efficiency. In 2018, Mobile DevOps was officially launched in the Apsara stack scenario. Currently, it has created value for dozens of key customers in multiple industries, enabling digital transformation of enterprise R & D processes. Public cloud free beta testing compared with Apsara stack, Mobile DevOps public cloud is more oriented to small, medium and micro enterprises. These customers have demands for improving R & D efficiency, but are sensitive to price. Public cloud is a good form of undertaking; at the same time, some external businesses (such as exclusive DingTalk) of Alibaba Group cannot be used for Mobile DevOps based on the group's internal R & D platform. Mobile DevOps public cloud is also a good choice. Since 2020.07, Mobile DevOps public cloud has officially launched a free public beta test. Currently, it has served a large number of small, medium, and micro-sized customers, as well as Alibaba Group's internal exclusive DingTalk, government DingTalk, and singing duck customers.

compared with Apsara stack, the Mobile DevOps of building a cloud-native form in the public cloud scenario faces more technical challenges. This chapter will share with you our thoughts on building a cloud-native Mobile DevOps, challenges and Solutions.

1. Why do I need public cloud Mobile DevOps?

1) provide inclusive Mobile DevOps services for small, medium, and Micro customers

although Apsara stack deployment has the advantages of exclusive access and internal network security isolation, the high cost of Apsara stack delivery is destined to be accepted by only high-end players in the industry. The cost of Apsara stack Mobile DevOps is evaluated as follows: • One-time investment: millions of one-time purchase costs • continuous investment: at least 30W/year server cost +20W/year Labor maintenance cost based on the above cost calculation, the investment costs of the first year, the second year and the third year of Apsara stack are respectively 1.5W, 50W, 50 million accumulates 200 million, which is unacceptable for small, medium and micro customers. As the infrastructure of the new era and the water, electricity and coal of the new era, it is necessary for Alibaba Cloud to provide inclusive cloud services for small, medium and micro enterprises other than major customers. The Mobile DevOps of the public cloud form is exactly in line with this concept. Based on the advantages of cloud-native elastic scaling and pay-as-you-go billing, the cost of using Mobile DevOps for small, medium, and Micro customers can be greatly reduced. In addition, in the public cloud scenario, the DevOps development process is more suitable for Target customers based on the characteristics of small, medium, and Micro customers.

2) integrate the EMAS product line to provide developers with a one-stop mobile R & D platform.

The launch of public cloud Mobile DevOps can effectively link EMAS with existing Mobile testing, Mobile monitoring, Mobile hotpatch, and other products, allowing EMAS to cover the entire lifecycle of applications, upgrade EMAS from mobile middleware to mobile R & D platform to improve user experience and stickiness. Compared with traditional self-built CI/CD platforms based on open-source solutions such as Jekins and Gitlab Runner, EMAS one-stop mobile R & D platform has obvious advantages in terms of cost, high availability, and technical support, in addition, it can cover the entire lifecycle management of application construction, testing, release, operation and maintenance, and Operation. Compared with the traditional self-built CI/CD "chimney" independent open source systems, R & D collaboration also has obvious advantages.

2. Challenges faced by public cloud Mobile DevOps

compared with Apsara stack internal network deployment and internal staff usage, Mobile DevOps in the public cloud will face more technical challenges, mainly in the following aspects:

1) Security

• Tenant isolation the first problem facing the public cloud is tenant isolation. Different customers need to use shared resources at the same time and cannot see each other's data. In this scenario, the building tasks of different customers may affect each other, and the building environment also involves users' private information such as code and certificates, A comprehensive solution must be provided to ensure the isolation of the user Building Environment • The security of private data such as code, certificate, and key. If you build a private data, it must involve user code, certificate, and key, these data are extremely private data. Any problems in the storage, transmission, and use of public cloud may cause significant losses to users. • External attacks because public clouds are exposed to the Internet and can be used by anyone, they also face the risk of malicious hacker attacks. In particular, building a public cloud involves a large number of custom execution commands, A complete mechanism must be provided to prevent hackers from running malicious custom commands and leaving backdoors in the build environment.

2) High Availability

• When elastic scaling is required for the growth of public cloud services, you must be able to quickly scale your services to adapt to the growth of your services. Otherwise, service exceptions may occur. This requires that cloud products conform to the distributed architecture in terms of technical implementation. In particular, cluster building must support stateless and rapid scale-out. • Stability of the build environment the build environment must be stable to avoid damage to the build environment caused by attacks or unusual use, such as environment variables and build toolchains. • High-standard SLA, real-time online, never downtime. High-standard SLA is not only a commitment to customers, but also a reverence for Alibaba Cloud brands.

(3) Scalability

• Differences in construction processes caused by diversified application architectures the number of junior cloud customers is limited, and there are perfect KA customer technical support services. Therefore, there are limited differences in applications and special personnel to support access. However, there are many customers in the public cloud environment, and the diversity of application architectures puts forward higher requirements for the universality and scalability of the system. • Diversified R & D processes the R & D team size, R & D culture, and R & D process of different customers in the public cloud are different, which also puts forward higher requirements for the scalability of Mobile DevOps R & D processes.

3. Our solution

in response to the challenges faced by the above public cloud Mobile DevOps, we will solve them through technical means from the following two aspects:

1) general architecture based on pipeline

the pipeline architecture implements generic construction, customizes the orchestration and construction process based on the pipeline, and extends the pipeline business capabilities based on the task plug-in, which solves the preceding scalability problems. This architecture has the following features: • General Architecture, supporting platform-wide building capabilities • custom orchestration and construction process based on YAML • visual orchestration of the pipeline • unlimited extension of task plug-ins supported by the pipeline

(2) build clusters based on containerization and virtualization

using the containerization (Linux)/virtualization (Mac OS) solution can completely solve various security and stability problems caused by resource sharing. Each build task starts a new container/virtual machine, containers and virtual machines are destroyed immediately after the build task is completed. This can not only effectively isolate the running environments between tasks, but also prevent the build environment from being damaged; in addition, a stable stateless containerization/virtualization cluster can be built to ensure the high availability of the build service. In the following three and four chapters, we will elaborate on these two points respectively and decrypt their design architecture and technical details.

1. Technology pre-research

in fact, there are not many friendly products based on assembly line design in the industry, especially many similar products abroad, such as Azure DevOps Pipeline and Github Actions, which are two excellent assembly line products, compared with other products, comprehensive consideration of usability, documentation and user scale has many advantages. Azure DevOps, formerly known as Visual Studio Team Services(VSTS), is a software R & D collaboration platform with a history of more than ten years. Its Azure Pipeline products were released in April 2018 [3]; github Actions product was released in August 2019 [4], which is a heavyweight product released after Microsoft acquired Github. In general, both belong to relatively new platforms, and the Azure Pipeline is only more than 2 years. An interesting phenomenon was found in the pre-research. As Github is already a subsidiary of Microsoft, the design concepts of the two pipeline products are not only similar, but also the Mac Virtualization solutions of the two are shared with each other in the pre-research, even Mac virtualization cluster data centers are shared. In terms of differences, Github Actions is simpler and more elegant than Azure Pipeline. In addition, Github Actions still continues the open-source Github style. Its Pipeline plug-ins are all open-source. Although it has only been online for more than one year, there have been more than 5,000 open-source plug-ins. From the perspective of plug-ins, this is a gold mine. If these plug-ins can be directly used in Mobile DevOps, the functional plug-ins of the basic pipeline will be aligned with the open source community. Considering the possibility of supporting these open-source plug-ins in the future, Mobile DevOps design architecture will also embrace the Github Actions of the open-source community.

2. Core concepts of pipeline

• Trigger a Trigger to Trigger a Pipeline. • Pipeline the Pipeline, which is the minimum unit to be triggered. A pipeline can contain one or more jobs • JobJob is the smallest unit to be scheduled. It can be divided into Agent (build cluster) and Agentless (server) according to the execution environment to which the Job is scheduled. Two types of jobs. Multiple jobs can run in parallel without dependencies or run in sequence with dependencies. The relationship between multiple jobs can be represented by a DAG. Each Job can contain one or more steps • Step

Step 是被执行的最小单位。每个Job由多个顺序执行的Step组成

• Task

Task是预定义规格和功能的任务插件，可以在Step中被声明引用执行，一个Step只包含一个Task

3. Technical architecture of pipeline

the pipeline consists of the following core systems:

1) Pipeline process engine

responsible for the triggering, orchestration, and state flow execution of the pipeline, as well as the maintenance of pipeline metadata information. The pipeline trigger module triggers the execution of a pipeline. It supports three trigger modes: manual, timer, and event (git event,webhook callback). Triggers are the only entry for pipeline execution. At this level, you can perform checksum checks by callers and pass in different trigger parameters to control pipeline execution and scheduling. Pipeline orchestration module pipeline orchestration defines a DSL language used to describe a pipeline. Based on this DSL language, a pipeline that can be scheduled and executed can be accurately defined. Pipeline execution module the pipeline execution module ensures that all jobs in the pipeline are executed in parallel or sequence according to the correct dependencies, and updates the real-time status of pipeline flow in real time.

2)Job scheduling engine

A Job is the smallest unit to be scheduled in a pipeline. The Job scheduling engine is responsible for scheduling each Job generated by the pipeline process engine to the correct cluster machine.

3) Integration Engine

there are two types of task plug-ins in the pipeline. One is Agent tasks, such as Android and iOS. These tasks require a specific build environment, therefore, it is natural to think that tasks will be scheduled to the builder by the Job scheduling engine. Another type of tasks are Agentless tasks, such as approval, notification, and external system calls, this type of task can be completed on a common server without the need to consume valuable build resources, and is scheduled to be executed by the Job scheduling engine on the integration engine. Most Agentless tasks are related to external service integration.

4)Channel Channel service

Channel Channel is used to build the communication link and protocol implementation between the cluster and the server. The main functions are as follows: • unified authentication of cluster building requests. For security reasons, the cluster building is in a different VPC from other microservices, complete network isolation ensures that the cluster cannot directly access the server intranet. Based on this background, the build cluster in the preceding pipeline architecture diagram accesses the server through the Internet HTTPS request, which requires authentication of the build machine request, channel Channel is the authentication server port • build a cluster request unified port build a cluster needs to keep heartbeat, status reporting, pull tasks, report task execution status with the server in real time, Channel the port of these requests, distribute requests from different businesses to different microservices.

(5) build a cluster

build a cluster is mainly responsible for pulling and executing Agent construction tasks. The services running in the build cluster are responsible for starting the isolated build environment that matches the task type: • Start Docker container Android on Linux. Build a Docker container based on Linux. The Docker container solution on Linux is the best choice for environment isolation. It is based on ACK serverless (Alibaba Cloud Public Cloud K8S products) start the serverless Docker container and automatically destroy and recycle the container. The cloud-native ACK serverless maximizes the elasticity of building clusters, and does not consume any computing resources, greatly controlling the construction cost. • Starting virtual machines on the Mac OS platform due to the limitations of the Apple ecosystem, iOS and Mac App can only be built in Mac OS systems. However, there is no mature Docker-like container solution available in the current Mac OS, finally, we implement environment isolation based on virtualization solutions. We have built a cloud-based Mac virtualization cluster to pool Mac physical resources and quickly scale up and down the cluster. This fully complies with cloud native concepts. Each time a virtual machine is created, a virtual machine is dynamically created from the Virtual Cluster. After the virtual machine is created, it is immediately destroyed. It is worth mentioning that Mac Virtualization clusters are our technical advantages. In the following chapter 5, we will Mobile DevOps the practice in the direction of Mac Virtualization clusters in detail.

at present, Mobile DevOps Mac virtualization cluster construction solutions are absolutely in the leading position in China. We may be the first DevOps platform built on iOS based on Mac virtual technology in China, there are almost no domestic manufacturers that support iOS construction. The essential reason is the limitation of Mac Virtualization Technology: traditional Mac physical bare metal construction can only be used in internal environments, public cloud services are not available at all. The Mac virtualization cluster building solution is Mobile DevOps technical advantage.

1. Select a virtualization solution

due to the kernel limitation of the Mac OS platform itself, the containerization solution Mac OS the platform is extremely immature at present, Mac OS the environment isolation of the platform is basically the only way to virtualization. Selection of virtualization types the following figure shows the two types of virtualization solutions. Both solutions are implemented based on Hypervisor. The comparison between the two solutions is as follows:

virtualization Solution 1:• host-free OS virtualizes VMS directly based on Hypervisor, with high resource utilization, and is more suitable for virtualization solutions of cloud services • requires higher hardware compatibility. Virtualization Solution 2:• Virtualize VMS based on Hypervisor on the OS of the host, which is more suitable for desktop users. • due to the host OS, hardware compatibility is better. Based on the consideration that we Mobile DevOps to provide public cloud services, selecting solution 1 can improve the resource utilization rate more effectively, and hardware compatibility can be avoided as long as appropriate hardware products are selected. Apple ecosystem security compliance issues Apple ecosystem is closed and has many security compliance restrictions. The Mac platform has the following legal compliance restrictions:

1.MacOS must run on Apple hardware 2. For commercial use, only one macOS instance is allowed to run on an Apple hardware

Compared with the above four virtualization solutions, only Solution 4 is compatible with Apple's ecological compliance and compatibility, and solution 4 is actually the virtualization solution 1 we chose in the previous section. Based on the above virtualization types and Apple's ecological security compliance and compatibility considerations, we finally chose the above solution 4.

2. Virtualized clusters of cloud architecture

to provide public building services on the cloud, virtualization solutions alone are not enough. A virtualized cluster solution that conforms to the cloud architecture is also needed to meet Mobile DevOps requirements for building clusters: ① Mac hardware resource pool-each Mac resource in the cluster should be stateless. All Mac hardware resources together form a resource pool, which can be uniformly allocated and scheduled by the cluster. (2) elastic scaling-the scale of public cloud business has certain elasticity, which requires that virtualized clusters can also adapt to business scenarios, and can quickly and flexibly scale up and down to keep up with the growth rate of business. (3) High Availability-in the case of a single Mac hardware device being damaged, the cluster can quickly and automatically respond to the task assignment to a new virtual machine to improve the success rate of task execution. From a single virtual machine to a virtual machine cluster, in addition to the preceding Mac hardware resource pooling, it also solves the newly introduced distributed storage and distributed network problems after hardware resource clustering, from a virtual single machine to a virtual cluster as shown in the following figure:

future Outlook

currently, the public cloud Mobile DevOps is still in the beta phase, and there are still many aspects to be worked on: • increase the ability to build intelligent analysis and prompt for errors. In the case of a large number of public cloud users, building incorrect Q & A is a huge labor cost. In the future, keyword matching, big data analysis, even technical means such as AI automatic error classification directly prompt the cause of construction errors, reduce the cost of manual Q & A • strengthen more linkage with other EMAS products, allows Mobile DevOps to integrate the entire application R & D lifecycle and maintain better affinity with the community. Supports Pipeline migration from Github Actions, Azure Pipeline, and other platforms to Mobile DevOps. Task plug-ins directly support Github Actions more than 5,000 open-source plug-ins, enjoying the benefits of the open-source community • enhancing integration capabilities, this allows Mobile DevOps Mobile R & D platform to be better integrated into customers' existing R & D processes • deeply optimize application compilation and construction efficiency and reduce application construction duration. The ultimate goal is to build applications on the cloud in a significantly shorter time than on-premises construction, allowing developers to intuitively feel the advantages of building on the cloud, or you are interested in the direction of cloud native, and you are a person who likes technical challenges. Welcome to join us. Our goal is to become an international leading mobile DevOps brand ".➡️ click here to view the position information.

References:

[1]Azure DevOps: What Is DevOps? [2] Baidu Statistics and Traffic Research Institute [3] Microsoft released Azure Pipelines, open-source projects can be used without restriction CI/CD[4] all open-source projects can be used free of charge, GitHub built-in CI/CD finally comes!