DevOps in the cloud native era-Alibaba Cloud Developer Community

DevOps is a cooperative process between software developers and IT personnel, with the goal of efficiently and automatically executing software delivery and infrastructure change processes. In the cloud native era, how can enterprises use DevOps to quickly, stably, efficiently, and securely iterate products and release business value.

What is cloud native?

To solve the problems such as slow upgrade of traditional applications, bloated architecture, inability to quickly iterate, failure locating, and failure to quickly solve problems, the concept of cloud native emerged.

Pivotal is the pioneer and Pathfinder of Cloud-native applications. It has launched Pivotal Cloud Foundry Cloud-native application platform and Spring open source Java development framework. As early as Pivotal 2015, the Matt Stine of the company wrote a pamphlet called migrating to cloud-native application architecture, which discussed several main features of cloud-native application architecture: application conforming to 12 factors, it is a microservice-oriented architecture, a self-service Agile Architecture, API-based collaboration, and anti-vulnerability.

Later, Pivotal updated the definition of cloud native several times. The latest introduction to cloud native applications on the official website of Pivotal focuses on the following four points: DevOps integration, continuous delivery, microservices, and containerization.

• DevOps is a collaboration between software developers and IT operations, with the goal of automating software delivery and infrastructure change processes. It creates a culture and environment in which software can be constructed, tested and released quickly, frequently and more reliably.
• Continuous delivery enables a single application change to be published after it is ready, without waiting for events such as bundling with other changes or waiting for maintenance window periods. Continuous delivery makes Publishing behavior insipid and reliable, so enterprises can deliver frequently with lower risks and obtain feedback from end users faster, until deployment becomes an essential part of business processes and enterprise competitiveness.
• Microservices are an architecture that develops applications as a small set of services. Each service can implement business functions, run in its own processes, and communicate through HTTP API. Each microservice can be deployed, upgraded, expanded, and restarted independently of other services in the application. It usually runs as part of the automation system, applications in use can be updated frequently without affecting end customers.
• Compared with standard virtual machines, containers provide both efficiency and speed. A single operating system instance uses operating system-level virtualization to dynamically divide one or more isolated containers. Each container has a unique writable file system and resource quota. The low overhead of creating and destroying containers and the high packaging density in a single virtual machine make containers a perfect computing tool for deploying microservices.

Google led the establishment of the Cloud Native Computing Foundation (CNCF), which defines cloud native as 1. Cloud Native technology helps enterprises and organizations build and run scalable applications in new dynamic environments such as public, private, and hybrid clouds. Cloud Native technologies include containers, service grids, microservices, immutable infrastructure, and declarative APIs. 2.l these technologies can build a loosely coupled system with good fault tolerance, easy management and easy observation. Combined with reliable automation, cloud-native technology allows developers to easily make frequent and predictable major changes to the system. CNCF is the biggest driver behind cloud native. Its key technologies include containers, microservices, service grids, devops, and declarative APIs.

Compared with traditional applications, cloud-native applications can take full advantage of the cloud, flexibly distribute applications to cloud vendors, release enterprise productivity, and focus on business innovation, instead of spending more time adapting and expanding different infrastructure platforms.

New challenges of DevOps in cloud native era

first of all, we need to know clearly that from the perspective of enterprises, in such an era of fast business, what do enterprises need most?

• Fast and unbroken. Here, we can interpret two meanings. One is that the quick launch of business applications is conducive to seizing market opportunities. The second meaning is that when your business grows explosively, how can you fully guarantee the computing resources? At this time, you may not be able to keep up with the rapid development of your business by adding a huge amount of IT investment to purchase hardware and software. This is actually the issue of enterprise R & D efficiency.
• Seek change in stability. Business or application stability always comes first. How to ensure business stability and meet the "sensitive" requirements of fast business, such as the launch of new business and application changes. This is the problem of enterprise IT architecture.
• Saving resources, how to save computing resources, and automatically scale up and down according to whether the business is in peak hours, this is the problem of cloud platform construction
• pioneering and innovative, developing O & M integration and microservice architecture

the initial appearance of DevOps broke the barriers and gaps between developers and O & M personnel, and strengthened the communication, collaboration and integration among developers, operations and quality assurance personnel. In the post-DevOps era, we can use container technology to iterate and launch applications faster.

The following is the general process of application release. Developers push the code to trigger the build. The build process is to pull the source code, package the application, push the container image, and deploy it.This model has actually there are many places made full use of cloud native advantage, such as a container technology, kubernetes, dynamic distribution slave pod, etc. But there are still some challenges.

• How to promote the security of applications between environment stacks
• how to manage application publishing permissions and security approval
• how to improve the average deployment time and recovery time of an application
• how to quickly locate, reproduce, and roll back online applications

DevOps in the cloud native era

first, we need to make full use of the advantages of cloud-native technology. Cloud-native can improve the efficiency of application development, change the organizational structure of enterprises, and even directly affect the decision-making of a company at the cultural level. In the container field, Kubernetes has become the community standard for container orchestration and management. It abstracts application services into multiple resource types, such as Deployment and Service, and provides a common portable model for cloud-native applications. In this context, how to implement more efficient DevOps in a cloud-native environment to achieve more productive performance has become a new topic and demand.

The following is the goal of building an enterprise application platform:

based on the PaaS platform, we designed a GitOps security release model to solve some of the challenges we mentioned earlier.

When designing a GitOps release model, the following core requirements are met:

• version Management. We hope that the version number of each published application can be associated with the git commit id. The advantage of this is that each change has a history query and can be quickly located and fixed.
• Baseline management. Easy problem reproduction and quick rollback
• secure release. Including publishing permission management and security approval;
• quick feedback. Improved R & D performance

the GitOps release model has the following features:

• Git repository is the only input source for any CICD process
• declarative application orchestration and deployment model building
• no difference between applications in environment stacks and automatic promotion
• pull pipeline process triggered by PR/MR
• fast feedback mechanism

the following figure shows the architecture of using GitOps to manage applications published to different kubernetes clusters. First, the application source code is separated from the build source code. We can see the two source code projects in the orange box. One is our application source code project application-java-demo, the source code project on the left is used to store the build source code, such as the Jenkinsfile of preview pipeline, the Jenkinsfile of staging pipeline, and the Jenkinsfile of production pipeline. In addition to Jenkinsfile, there may be some information about dynamically creating a test environment, sensitive information about the connection to the staging environment or production environment can also be stored in the database, where the connection information of the database is stored. The Common application-java-demo has different branches in the Git repository, and each branch has a certain correspondence with the kubernetes cluster environment. For example, in our settings, the master branch corresponds to the production environment, the latest branch corresponds to the pre-release environment, other development branches correspond to the test environment, the dynamic creation and destruction of the test environment, the deployment and release of the application retest environment is a self-service provided by developers and testers. However, if an application is to be deployed in the pre-release environment and the production environment, it must be approved by the administrator for security. Ordinary developers only have the permission to create new code branches and merge requests. In addition, the administrator has the permission to do the rest. The green area is Jenkins pipeline task, you can also use other cicd engines to build this pipeline task. Ordinary developers do not have the permission to create and build a Job Jenkins the environment, nor do they have the permission to change the configuration. They only have the permission to view logs of a Job.

The last is a sequence diagram showing the complete process of an application from development testing to business launch iteration: (1) the developer submits a new function branch feature;(2) the developer creates a Merge Request that requests to Merge code into the latest branch;(3) developers create Merge Request automatically trigger the construction of a Jenkins pipeline task named preview-pipeline (4) the preview-pipeline pipeline task pulls the preview-pipeline source code project from the Git server, and run the process of source code compilation, testing, container Image construction and push, application deployment to Jenkinsfile container cluster, Preview notification according to the declarative script in the DingTalk file in the project. (5) the administrator checks the preview connection of the application on the Merge Request page of The Git server and verifies whether the application can be merged into the latest branch. If the application passes the verification, the administrator accepts the Merge of the Merge Request and triggers step 6, if the code fails, the developer is notified to update and submit the code. Step 1(6) the administrator accepts the Merge action of the Merge Request, which will automatically trigger the Jenkins pipeline task staging-pipeline construction (7) the staging-pipeline pipeline task pulls the staging-pipeline source code project from the Git server, and runs the source code compilation, testing, container Image construction, and push according to the declarative script in the Jenkinsfile file in the project, the process of deploying applications to Staging container clusters and DingTalk notifications. (8) after the application service in Staging environment passes the test and verification, the administrator can merge the latest branch into the master branch (9) after the administrator merges the latest branch into the master branch, jenkins pipeline task production-pipeline construction will be triggered automatically (10)production-pipeline pipeline task pull production-pipeline Source Code Project from Git server, and run the process of source code compilation, testing, container Image construction and push, application deployment to Jenkinsfile container cluster, Production notification according to the declarative script in the DingTalk file in the project.

GitOps is a set of methodologies, so there are many practical methods and a variety of useful tools. For example, using draft can help automate the generation of application orchestration templates, skaffold is used to simplify the application construction and deployment process. kaniko can build and push images that do not depend on docker daemon. helm is used as a package management tool for applications. There are other cicd engines such as jenkins,tekton,argo and jenkinsx for Cloud Native

later, we will demonstrate how the GitOps security release model works separately.

References: https://pivotal.io/cn/cloud-native