Alibaba Cloud Container-based CI/CD implementation practices-Alibaba Cloud Developer Community

Author's personal introduction Liu Chen Lorraine

coordinate Fintech, proficient in continuous integration and release, with experience in continuous deployment and release of more than 100 applications across the platform, and now aspire to become a K8S player.

Backdrop

hello everyone, my team is currently facing the major task of digital transformation of the company's business. The main R & D challenge we are facing is how to quickly iterate new development requirements. Because there is not much historical burden, the technology stack chosen by the team is relatively mature and popular, for example, the Devops I am responsible for is mainly built and implemented based on the Alibaba Cloud ACK container and Jenkins2.0 +.

The Alibaba Cloud ACK container is also a cloud service encapsulated in K8S1.16.9. We chose the managed version, that is, the master node is hosted in Alibaba Cloud. We are only responsible for the O & M and management of the worker node cluster. The advantage of this is that the team should focus on the business level as much as possible, and the operation and maintenance work at the infrastructure level should be served as much as possible. Jenkins2.0 has many advantages. For example, Jenkins is a classic CI implementation tool platform. Most developers and testers are familiar with this method and do not need to learn again. After version 2.0, declarative Jenkinsfile syntax has been introduced. This pipeline orchestration format can be more naturally integrated with K8S based on declarative API, and the plugin of Jenkins ecosystem is very rich, it can basically meet the needs of team building multi-language and multi-project CICD tasks through configuration.



core issues

currently, the following core issues must be solved in the implementation of CICD services.

What is the deliverable?

What are the deployment environments? What is the environment configuration information? How do I automatically perform build and deploy tasks? How do I detect faults if the build and deployment tasks fail, or applications in the deployment environment fail to run?

Definition and structure of deliverables

deliverables are the objects that need to be determined first in the entire CICD process. They are the objects that interact with devops. Developers and devops are involved in the definition of deliverables, and devops provides a common template for deliverables based on the definition.



deliverables definition

the team defines deliverables as microservice images generated based on dockerfile.

devops provides a private repository based on Alibaba Cloud Container Registry, as shown in the following figure: the development provides Image Packaging Definitions based on dockerfile. Take a java application as an example. The image content mainly includes the working directory based on the openjdk basic image and the specified startup cmd. The sample code is as follows:

FROM openjdk:11.0.8WORKDIR /home/demoCOPY target/index-0.0.1-SNAPSHOT.jar app.jarENTRYPOINT ["java","-jar","app.jar"]

deliverable structure

each application deployment delivery object is archived in an independent image repository. The image repository is named as the project name. The name of the application service. The naming convention of the image tag should include the code changes contained in each build object, which are represented by the latest commit id, and the time when the build occurred, the timestamp is represented in the yyyymmddd-hhmm format. The example is as follows:

the deliverable image shall be an entity that contains all information about application deployment except environment configuration information. Take a java application as an example. The delivery object is an image entity of an openjdk-based jar package. Environment configuration information can be passed and rewritten by environment variables.



deployment environment and configuration

deployment environment

one of the core issues that CICD solves is how to deploy the same delivery object in multiple environments. There are four typical deployment environments:

dev development environment

the environment in which the deliverables are generated. In a development environment, the first time a deliverable is generated, it requires necessary steps such as unit test pass rate and code security check. The deliverable in this environment is a measurable deployment.

test Environment

the environment in which deliverables start integration testing and regression testing. Deliverables in this environment are deployments that can be produced.

stage pre-release environment

the scenario of the pre-release environment varies according to requirements. Some companies use the pre-release environment as the demo environment as the poc function display; Some companies use the pre-release environment for traffic stress testing, make sure that the service load before production matches the resource allocation. Some companies use the stage environment to switch traffic when doing blue-green release or A/B test. Regardless of how the stage environment is used, the configuration of the stage deployment environment must be consistent with that of the production deployment environment.

prod production environment

the production and release environment is the final runtime environment of deliverables, providing services to users.

Environment configuration

environment configuration information is a series of environment variables that are independent of deliverables during CICD but depend on the deployment environment. When you deploy a Kubernetes cluster, you can ConfigMap and Secret resource objects.

Build automation

we have Dockerfile to describe deliverables definition of simple as long as docker build and then docker push to private mirror warehouse completed build. An automated build task based on Jenkins automates the process. The implementation part will be described in detail later.



the following are the main considerations for building task automation:

how do I automatically trigger the execution of the build task when the main/master branch has code commit generation?

What is the construction process for applications in different languages, such as java applications or front-end applications? What are the steps?

How can the generated delivery image be successfully stored in the private repository of Alibaba Cloud Container Registry?

How does the delivery process ensure the security and reliability of network transmission?

Deployment Automation

our deployment environment is in an Alibaba Cloud Container cluster. Each deployment environment is isolated by namespaces, and will be upgraded later. Currently, the mainstream solutions for application deployment in Kubernetes clusters are Helm and Kustomize. Finally, we chose HelmV3 for implementation. The choice between Helm and Kustomize is another interesting topic, which is not discussed in this chapter. Helm-based deployment automation is simply to build a jenkins task that can run helm Install commands. The implementation part will be described in detail later.



deployment Automation mainly considers the following issues:

how do I obtain deliverables?

Which deliverables are deployed? Which environment is the target?

How do I define a successful deployment?

Deployment failed. How do I detect the failure? How to quickly locate a problem?

How do I roll back?

Event monitoring and alerting mechanism

CICD aims to automate the whole process as much as possible to reduce human participation. The higher the degree of automation, the more necessary it is for automatic task fault discovery and alarm. Our services are deployed in Alibaba Cloud container clusters. Based on Jenkins2.0, we build automated CI/CD tasks, build deployment dependencies, and Jenkins whether the tasks are successfully executed, and whether the resources in the cluster are updated as scheduled according to the definition of Helm. CI/CD-related event monitoring focuses on cluster resource Event Monitoring and Jenkins task monitoring.

Alibaba Cloud Container cluster event monitoring and alerting

alibaba Cloud provides event monitoring and alerting services, which can be directly configured and used. As shown in the following figure, this example provides event monitoring dashboard services for Alibaba Cloud cluster services, O & M management, and event lists. We can easily obtain the running status of the entire cluster resources.

Log on to the alert configuration service to configure alerts for basic events. The cluster initializes alert configurations for many basic events, such as Pod/Node OOM. Pod startup fails or resources are insufficient, unable to schedule. You can customize alerts based on your business needs. Currently, the author uses an alert template configured by default to tolerate the occurrence frequency and send alerts by email. Jenkins an automatic email alert when the task fails.

E-mail Notification Plugin can help realize the automatic email sending function of Jenkins service for task completion.

First in Jenkins/configuration/Extended E-mail Notification configuration SMTP mail service information, we are using Alibaba Cloud mail push service, make sure to fill in the correct smtp server, smtp port, if with smtp username/passwrod, you must also enter the correct information. In the Jenkinsfile, the email extension plugin is declared as follows:

emailext (

            to: 'XXX@example.com',
            subject: "FAILED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'",
            body: """<p>FAILED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
            <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>""",
            recipientProviders: [[$class: 'DevelopersRecipientProvider']]
        )

#### Implementation

build CI pipeline based on Dockerfile/Jenkinsfile

deliverables are defined by Dockerfile. Each application service must have a Dockerfile file under its root directory, which defines the service building process. This section describes java microservices, which is also the build pipeline of our back-end microservices.

Java microservices

Java application building logic is defined in Jenkinsfile and implemented based on maven. Dockerfile template only defines how to call the built jar package, that is, the deployment instruction. The main purpose of this is to minimize the image and reduce the load of network data transmission.

The advantage of declarative Jenkinsfile is that it can define and manage pipeline logic through code. You can manage versions based on code.

To build a Pipeline, follow these steps:

scm obtains project source code

build a maven package to generate a jar package that can be deployed.

maven-based unit testing

sonarQube-based code check

generate an image based on the cloud native mode of kaniko and push it to the Alibaba Cloud Private image repository

in addition to the main process of building Pipeline, the subsequent steps are the downStream tasks after the construction task is completed. The links are general deployment tasks. This structure decouples the construction and deployment logic, different build tasks can reuse the same deployment task.

def branch_name

def revisiondef registryIp = "registry-vpc.cn-shanghai-finance-1.aliyuncs.com"def app = "XXX"pipeline {

agent{
    node{
        label 'slave-java' // Jenkins Slave Pod Template
    }
}
stages {
    stage ('Checkout') {
        steps {
            script {
                def repo = checkout scm
                revision = sh(script: 'git log -1 --format=\'%h.%ad\' --date=format:%Y%m%d-%H%M | cat', returnStdout: true).trim()
                branch_name = repo.GIT_BRANCH.take(20).replaceAll('/', '_')
                if (branch_name != 'master') {
                    revision += "-${branch_name}"
                }
                sh "echo 'Building revision: ${revision}'" // 获取代码并生成镜像tag(latest-commit-id+timestanp+branch)
            }
        }
    }
    stage('Compile') {
        steps {
            container("maven") {
                sh 'mvn -B -DskipTests clean package' // 基于maven的构建步骤
            }
        }
    }
    //unit test 测试部署
    stage('Unit Test') {
       steps {
           container("maven") {
               sh 'mvn test org.jacoco:jacoco-maven-plugin:0.7.3.201502191951:prepare-agent install -Dmaven.test.failure.ignore=true'
           }
       }
    }
    // 上传Jacoco检测结果
    stage('JacocoPublisher') {
        steps {
            jacoco()
        }
    }
    stage('Build Artifact') {
        steps {
             container("maven") {
                sh 'chmod +x ./jenkins/scripts/deliver.sh'
                sh './jenkins/scripts/deliver.sh'
             }
        }
    }
    stage('SonarQube Analysis'){
        environment {
            scannerHome = tool 'SonarQubeScanner'
        }
        steps {
            withSonarQubeEnv('sonar_server') {
                sh "${scannerHome}/bin/sonar-scanner"
            }
        }
    }
     // 添加stage, 运行容器镜像构建和推送命令
    stage('Image Build and Publish for Dev Branch'){
      when { not { branch 'master' } }
      steps{
          container("kaniko") {
              sh "kaniko -f `pwd`/Dockerfile -c `pwd` --destination=${registryIp}/xxxx/xxx.${app}:${revision} --skip-tls-verify"
          }
      }
    }
     // 添加stage, 运行容器镜像构建和推送命令
    stage('Image Build and Publish for Master Branch'){
      when { branch 'master' }
      steps{
          container("kaniko") {
              sh "kaniko -f `pwd`/Dockerfile -c `pwd` --destination=${registryIp}/xxxx/xxx.${app} --destination=${registryIp}/xxxx/xxx.${app}:${revision} --skip-tls-verify"
          }
      }
    }
}
post {
    always {
        echo 'This will always run'
    }
    success {
        script {
            build job: '../xxx.app.deploy/master', parameters: [string(name: 'App', value: String.valueOf(app)), string(name: 'Env', value: 'dev-show'), string(name: 'Tag', value: String.valueOf(revision))]
        }
    }
    failure {
        emailext (
            to: 'XXX@example.com',
            subject: "FAILED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'",
            body: """<p>FAILED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
            <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>""",
            recipientProviders: [[$class: 'DevelopersRecipientProvider']]
        )
    }
    unstable {
        echo 'This will run only if the run was marked as unstable'
    }
    changed {
        echo 'This will run only if the state of the Pipeline has changed'
        echo 'For example, if the Pipeline was previously failing but is now successful'
    }
}

}

generally, a build task is automatically triggered based on the code submitted by the main branch. To achieve autoTrigger, we need to use webHook to connect SCM services and Jenkins services. The SCM Service we use is Bitbucket, which implements autoTrigger by using the hook plug-in.

The deployment environment based on Alibaba Cloud container service belongs to the VPC, that is, the Container Registry we use is a private image service in the VPC, which cannot be accessed outside the VPC. The Kubernetes cluster also belongs to the resources in the VPC. Therefore, the CI Jenkins Slave Pod in the cluster and the Pod service deployed in the target cluster are all in the internal network of the VPC and are directly connected to the private Container Registry, the security and reliability of data transmission are ensured.

Build a CD pipeline based on Helm/Jenkinsfile

to deploy an application on K8S, kubectl is typically used to create a series of resource objects (such as deployment,configMap,secret,serviceAccount, service, and ingress). In the CICD deployment process, the same deployment object is deployed to multiple environments, that is, multiple deployments occur. For example, the structures of the two Deployment objects deployed in the development and test environments are basically the same, and only a few attribute values vary depending on the environment. An important concept in CICD software practice is that deployment can be repeated to ensure that deployment actions are consistent in each environment to avoid introducing differences in the release pipeline. Helm is a template solution to meet the above requirements. The introduction of Helm is not included in this Chapter. Readers who are interested in learning can refer to it. https://whmzsu.github.io/helm-doc-zh-cn/

helm provides a chart package (a collection of template files that can encapsulate K8S resource objects) and values.yaml (a collection of attribute parameters) structure, based on the Go Template syntax, decouples the property values in the manifest from the K8S resource object Template structure. This enables the creation of a set of resource objects required for application deployment to be reused in multiple environments and deployed repeatedly through template definition and assignment. This solves the problem of K8S resource object management.

The following uses the helm/Chart package of Java microservices as an example to illustrate how we generate application deployments.

You can use helm create Name [flags] to initialize helm packages.

The directory structure of the deployment helm package is as follows: the values in the root directory. yaml is a common set of properties in all environments. For example, for creating serviceAccount and rbac properties, dev/test/stage/production.yaml is a collection of properties related to the deployment environment.

Run the helm install Command to deploy the current Helm package template. For example

 helm upgrade index ./helm/ -i -nbss-dev -f ./helm/values.yaml -f ./helm/dev.yaml --set 'image.tag=latest' --set 'image.repo=bss.index' --set 'ingress.hosts.paths={/index}'

as mentioned above, we use declarative Jenkinsfile syntax to build deployment pipeline logic. Unlike traditional groovy, we can declare pipeline,agent/node,stages/step, and other objects, declare the definition of the deployment process. Take deploying a helm-based jar package application as an example. The deployment object is an image file in the Alibaba Cloud Private image repository, as shown in the preceding example. The deployment process includes two steps: 1. Obtain the helm code 2. Run the helm install command as follows:

def branch_namedef revision

def registryIp = "registry-vpc.cn-shanghai-finance-1.aliyuncs.com"pipeline {

agent{
    node{
        label 'slave-java' // Jenkins Slave Pod Template
    }
}
parameters {
    choice(name: 'App', choices: ['111', '222', '333','444'], description: '选择部署应用')
    choice(name: 'Env', choices: ['dev', 'stage', 'test',], description: '选择部署环境')
    string(name: 'Tag', defaultValue: 'latest', description: '请输入将要部署的构建物镜像Tag')
}
stages {
     stage ('CheckoutHelm') {
        steps {
            script {
                def repo = checkout scm
                revision = sh(script: 'git log -1 --format=\'%h.%ad\' --date=format:%Y%m%d-%H%M | cat', returnStdout: true).trim()
                branch_name = repo.GIT_BRANCH.take(20).replaceAll('/', '_')
                if (branch_name != 'master') {
                    revision += "-${branch_name}"
                }
            }
        }
    }

stage ('Deploy') {

         steps {
            container('helm-kubectl') {
                 sh "chmod +x ./helm/setRevision.sh"
                 sh "./helm/setRevision.sh ${revision}"
                 sh "helm upgrade -i ${params.App} ./helm/ -nxxx-${params.Env} -f ./helm/${params.Env}.yaml --set-file appConfig=./appConfig/${params.Env}/${params.App}.yml --set image.tag=${params.Tag} --set image.repo=bss.${params.App} --set ingress.hosts.paths={/${params.App}}"
            }
        }
    }
}

post {

    always {
        echo 'This will always run'
    }
    success {
        echo 'This will run only if successful'
    }
    failure {
        emailext (
            to: 'XXX@example.com',
            subject: "FAILED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]'",
            body: """<p>FAILED: Job '${env.JOB_NAME} [${env.BUILD_NUMBER}]':</p>
            <p>Check console output at &QUOT;<a href='${env.BUILD_URL}'>${env.JOB_NAME} [${env.BUILD_NUMBER}]</a>&QUOT;</p>""",
            recipientProviders: [[$class: 'DevelopersRecipientProvider']]
        )
    }
    unstable {
        echo 'This will run only if the run was marked as unstable'
    }
    changed {
        echo 'This will run only if the state of the Pipeline has changed'
        echo 'For example, if the Pipeline was previously failing but is now successful'
    }
}

}

experience Summary and prospect

self-help pipeline improvement

the business architecture that my team is facing is the growing microservice clusters. Based on this, new challenges are constantly posed to the automation requirements and release frequency of CICD. Originally, each microservice has an independent helm Deployment Package. However, due to the rapid increase in the number of microservices, for the rapid registration of new services into the existing CICD pipeline, the development of Quick Build and deployment services is required.

The main problem is that the helm deployment template is based on the resource object construction required for application deployment and is configured in different deployment environments. Each application creates a helm deployment package and independent build and deployment tasks. As more and more microservices need to be deployed, the workload of constructing helm packages and corresponding construction and deployment tasks becomes the efficiency bottleneck of CICD process.

The solution is to abstract a common helm deployment package template, abstract the custom property information set during application deployment from the original helm package structure, and separate the configuration information into a layer of AppConfig File, example: the abstract implementation of the template here mainly depends on the-f input values in the Helm command. In yaml, multiple property set files can be-f at the same time, and the files with-f behind can overwrite the previous property parameter values. If the custom configuration file is not a format that can be directly used by Go template, you can use flag -- set-file to directly write the content in the Config file to some upper-layer property parameters.

Security issues

access Security is an important issue that cannot be ignored when the development and test environments are moved to the public cloud. Our deliverables, deployment pipelines, and target deployment clusters are all based on a VPC. From the perspective of network transmission, data interaction is in a LAN, which is a relatively secure public cloud network environment. In addition, to ensure that the development environment and the Test environment are completely privatized, we also set corresponding access control on the slb corresponding to the ingress resource deployed in microservice applications, network access is only available to users within the company. In the future, KMS and other confidential data transmission services provided by Alibaba Cloud will be used to ensure data transmission between services accessed through the public network and between external deployment environments and CICD services.

Selected, One-Stop Store for Enterprise Applications
Support various scenarios to meet companies' needs at different stages of development

Start Building Today with a Free Trial to 50+ Products

Learn and experience the power of Alibaba Cloud.

Sign Up Now