Building Continuous Deployment on Containers and Best Practices

Date: Oct 25, 2022

Related Tags：1. Using Git with Jekyll on Alibaba
2. How to Select a Git Branch Mode?

Abstract: To understand continuous integration and continuous deployment, you must first understand its components and the relationship between them.

Continuous Deployment

As shown in the figure, the development process is as follows:

The programmer downloads the source code from the source code library (Source Control), writes the program, submits the code to the source code library after completion, and the Continuous Integration tool downloads the source code from the source code library, compiles the source code, and submits it to the runtime library ( Repository), and then the Continuous Delivery (Continuous Delivery) tool downloads the code from the runtime (Repository), generates a release version, and publishes it to different runtime environments (such as DEV, QA, UAT, PROD).

In the figure, the part on the left is continuous integration, which is mainly related to development and programmers; the part on the right is continuous deployment, which is mainly related to testing and operation and maintenance. Continuous Delivery (Continuous Delivery) is also called Continuous Deployment (Continuous Deployment). This article focuses on continuous deployment.

Continuous integration and deployment has the following main players:

Source code repository: responsible for storing source code, commonly used are Git and SVN.

Continuous integration and deployment tools: responsible for automatically compiling and packaging and storing runnable programs into runnable libraries. The more popular ones are Jenkins, GitLab, Travis CI, CircleCI, etc.

Repository Manager: The Repository in the figure, also known as the runtime library, is responsible for managing program components. The most commonly used is Nexus. It is a private library and its role is to manage program components.

The library manager has two functions:

Manage third-party libraries: Applications often use many third-party libraries, and different technology stacks require different libraries. They are often stored in third-party public libraries, which are not very convenient to manage. Generally, companies will establish a private management library to centrally manage various third-party software. For example, it can be used as a Maven library (Java), a mirror library (Docker), and an NPM library (JavaScript) , to ensure the standardization of the company's software.

Manage the delivery of internal programs: All the programs released by the company in various environments (such as DEV, QA, UAT, PROD) are managed by it, and given a unified version number, so that any delivery can be documented, and it is convenient to The program rolls back.

Continuous Deployment Steps

Different companies have different requirements for continuous deployment, and its steps are also different, but mainly include the following steps:

Download source code: Download source code from a source code repository such as github.

Compile code: This step is required for compiled languages

Test: Test the program.

Generate image: There are two steps here, one is to create the image, and the other is to store the image to the image repository.

Deploy image: Deploy the generated image to the container

The above process is a broad continuous deployment process, and a narrow process is to retrieve the executable program from the library manager, which saves the link of downloading the source code and compiling the code, and downloads the executable program directly from the library manager instead. But since not every company has a separate library manager, a broad continuous deployment process is used here, so that it works for every company.

Continuous deployment instance

Below we use a specific example to show how to complete continuous deployment. We use Jenkins as a continuous deployment tool to deploy a Go program to the k8s environment.

Our process is basically the narrow process mentioned above, but because there is no Nexus, we have changed it a bit and downloaded the source program directly from the source code library. The steps are as follows:

Download the source code: Download the source code from github to the running environment of Jenkins

Test: This step has no actual content for the time being

Generate an image: Create an image and upload it to Docker hub.

Deploy image: Deploy the generated image to k8s

Before creating a Jenkins project, do some preparatory work:

Create a Docker Hub account

You need to create an account and image repository on Docker Hub in order to upload images. The specific process is not explained in detail here, please refer to the relevant information.

Create Credentials on Jenkins

You need to set the user and password to access the Docker hub, which can be referenced by variables in the Jenkins script later, so that the password will not appear in the program in clear code.

After logging in to the Jenkins main page with an administrator account, find Manage Jenkins->Credentials->System->Global Credentials->Add Credentials, enter your Docker Hub username and password as shown below. "ID" is what you will refer to later in the script.

Create a Jenkins image with Docker and k8s pre-installed

There is no Docker and k8s in the default container of Jenkins, so we need to recreate a new image based on the Jenkins image, which will be explained in detail later.

Below is the image file (Dockerfile-modified-jenkins)

The above image installs Docker and kubectl on the basis of "jenkins/jenkins:lts", which supports these two software. The 19.03.4 version of docker is used in the image. This installs just the "Docker CLI", no Docker engine. When using it, it is still necessary to mount the volume of the virtual machine to the container and use the Docker engine of the virtual machine. Therefore, it is best to ensure that the Docker version in the container is the same as the Docker version of the virtual machine.

Use the following command to check the Docker version:

For details, see Configure a CI/CD pipeline with Jenkins on Kubernetes: https://developer.ibm.com/tutorials/configure-a-cicd-pipeline-with-jenkins-on-kubernetes/

The preparations have been completed, and now it is time to officially create the Jenkins project:

Jenkins script:

The creation of the project is done on the homepage of Jenkins, its name is "jenkins-k8sdemo", its main part is the script code, it is also stored in the same source library as the Go program, and the file name is also "jenkins-k8sdemo" k8sdemo". The script page of the project is shown in the figure below.


If you are not familiar with installing and creating Jenkins projects, see Installing Jenkins on k8s and FAQs

The following is the jenkins-k8sdemo script file:

Let's look at the code piece by piece:

Set container image:

Here, the container image of the Jenkins sub-node Pod is set, using "jfeng45/modified-jenkins:1.0", which is what we created in the previous step. All script steps (stages) use this image. "volumes:" is used to mount volumes into the Jenkins container so that Jenkins child nodes can use the virtual machine's Docker engine.

See jenkinsci/kubernetes-plugin for Jenkins script commands and setting mount volumes

https://github.com/jenkinsci/kubernetes-plugin

Create mirror:

The following code generates the Docker image file of the Go program. Here we do not use the Docker plug-in, but directly call the Docker command. Its benefits will be discussed later. It references the "Docker hub" credentials we set up earlier to access the Docker repository. In the script, we first log into the "Docker hub", then use the source code downloaded from GitHub in the previous step to create the image, and finally upload the image to the "Docker hub". in

WORKSPACE" is a predefined variable in Jenkins, and the source code downloaded from GitHub is stored in "{WORKSPACE}".

If you want to learn more about Jenkins commands, see Set Up a Jenkins CI/CD Pipeline with Kubernetes

We did not regenerate the image file of the Go program here, but reused the image file of the previous k8s to create the Go program.

Its code is as follows. The benefits of doing so will be discussed later.

For details on Go image files, see Creating Optimized Go Image Files and Stepped Pit

Deploy image:

Next, deploy the Go program to k8s. There is no kubectl plug-in here, but directly use the kubectl command to call the existing k8s deployment and service configuration files (the generated Go image will be referenced in the file), and its benefits will be discussed later. arrive.

For details on k8s deployment and service configuration files, see What do I need to modify to migrate an application to k8s?

Why not use Declarative?

There are two ways to write Pipeline with script, "Scripted Pipleline" and "Declarative Pipleline", here is the first method. "Declarative Pipleline" is a new method, the reason why I didn't use it is because I started using Declarative mode but didn't call it up, then I switched to "Scripted Pipleline" and it worked. Only later did I find out how to set up Declarative, especially how to mount volumes, but after reading it, it was much more complicated than "Scripted Pipeline", so I was lazy and didn't change it.

If you want to know how to setup mounted volumes in Declarative mode, see Jenkins Pipeline Kubernetes Agent shared Volumes

Automated items:

Projects in Jenkins now need to be started manually. If you need to start the project automatically, you need to create a webhook. Both GitHub and dockerhub support webhook, and there are setting options on their pages. "webhook" is a reversed URL, which is called by GitHub and dockerhub whenever there is a new code or image submission. The URL is set to the project address of Jenkins, so that the related project will be automatically started.

Test results

Now that the Jenkins project is fully configured, you need to run the project and check the results. After starting the project,

Check "Console Output", the following is part of the output (all the output is too long, please see the appendix), indicating that the deployment is successful.

Jenkins Principle

The examples section is over, let's explore best practices. Before that, we must first understand the principle of Jenkins.

Executable commands

One question I've always had is which commands are those commands that Jenkins can execute through the shell? Jenkins is different from Docker and k8s, which have their own set of commands, as long as you learn them. Jenkins works by integrating with other systems, so its executable commands are related to other systems, which makes it difficult to know which commands are executable and which are not. You need to understand how it works to get the answer. When Jenkins executes the script, the master node automatically generates a child node (Docker container), and all Jenkins commands are executed in this container. So the commands that can be executed are closely related to the container. Generally speaking, you can run Linux commands through the shell. Then the following question arises:

1. Why can't I use Bash?

Because the container you're using on the child node may be using a stripped-down version of Linux, such as Alpine, which doesn't have Bash.

2. Why can't I run Docker commands or Kubectl?

Because its default container is jenkinsci/jnlp-slave, and it doesn't have Docker or kubectl pre-installed in it. Instead of using the default container, you can specify your own container and pre-install the above software in it, then you can execute these commands.

How to share files

A Jenkins project is usually divided into several steps (stages) to complete. For example, the source code you downloaded needs to be shared among several steps. How to share it? Jenkins allocates a WORKSPACE (disk space) for each project, which stores all files downloaded from the source repository and other places, and files can be shared between different stages through WORKSPACE.

For details on WORKSPACE, see Jenkins Project Artifacts and Workspace

Best Practices

To summarize best practices, it is necessary to understand the role and position of continuous deployment in the entire development process. It mainly plays a role in connecting all links. The deployment of the program is completed by k8s and Docker, so the scripts for program deployment are also in k8s and maintained by k8s. We don't want to maintain a similar set of scripts in Jenkins, so the best way is to compress Jenkins scripts to a minimum and directly call k8s scripts as much as possible.

In addition, if you can write code, do not configure it on the page. Only the code can be repeatedly executed and ensure stable results. The page configuration cannot be ported, and it cannot be guaranteed that each configuration will produce the same result.

Use as few plugins as possible

Jenkins has many plugins, basically what you want to accomplish has a corresponding plugin. For example, if you need to use the Docker function, there is the "Docker Pipeline" plugin, and if you want to use the k8s function, there is the "kubectl" plugin. But it will bring a lot of problems.

First, each plugin has its own setup (usually on the Jenkins plugin page), but this setup is not compatible with other continuous deployment tools. If you later migrate to other continuous deployment tools, these settings will need to be discarded.

Second, each plugin has its own command format, so you need to learn another set of new commands.

Third, these plugins tend to only support partial functionality, limiting what you can do.

For example, you need to create a Docker image file, the command is as follows, it will create an image named "jfeng45/jenkins-k8sdemo", the default file of the image is the Dockerfile in the root directory of the project.

But the command to create a Docker image file has many parameter options. For example, your image file name is not Dockerfile, and the directory is not in the project root directory. How to write it? This was not supported in previous versions, but is supported in later versions, but after all, it is not very convenient, and new commands have to be learned. The best way is to use Docker commands directly, which perfectly solves the three problems mentioned above. The answer lies in the Jenkins principle mentioned above. In fact, most plug-ins are not needed. You only need to create a Jenkins child node container and install the corresponding software to solve it successfully.

Below is a comparison of the script with and without the plugin, the unused script looks longer because the script with the plugin has better integration with the credential settings in Jenkins, and the unused script does not. But aside from this small disadvantage, scripts that are otherwise not used are far superior to those that use plugins.

Script using plugin (with plugin command):

Script without plugin (direct Docker command):

Use k8s and Dcoker as much as possible

For example, if we want to create an image of an application, we can write a Docker file and call this Docker file in the Jenkins script to create it, or we can write a Jenkins script and create the image in the script. A better way is the former. Since both Docker and k8s are de facto standards, porting is easy.

The less code the Jenkins script has, the better

If you agree with the first two principles, then this one is logical, and the reason is the same as above.

FAQ:

1. Variables should be enclosed in double quotes

Jenkins scripts can use either single or double quotes, but if you quote variables in quotes, use double quotes.

2.docker not found

If there is no Docker in the Jenkins container, but you call the Docker command, the following error will appear in the "Console Output":

3.Jenkins is down

When debugging Jenkins, I created a new image file and uploaded it to "Docker hub" and found that Jenkins was down. Checked the Pod and found the problem, k8s can't find the Jenkins image file (the image file disappeared from the disk). Because the Jenkins deployment file is set to "imagePullPolicy: Never", once the image is gone, it will not be automatically re-downloaded. Later, I found the reason. The default disk size of Vagrant is 10G. If the space is not enough, it will automatically delete other image files from the disk to free up space. As a result, the image file of Jenkins was deleted. The solution is to expand the disk of Vagrant. size.