Implementing an Optimized DevOps (CI/CD) Environment Using Alibaba Cloud DevOps Flow

By Jongjin Lim, Solution Architect at Alibaba Cloud

1. Background Description

As a business expands, the diversity and complexity of its IT environment also increases. Even within the same company and the same team, different platforms, tools, and cloud environments are being used depending on the nature of the work.

Despite this, many companies still maintain legacy development environment and processes, which are not well equipped to deal with such issues. As a consequence, companies oftentimes are not able to effectively utilize innovative platforms even if the company invested heavily in such technology.

DevOps is a process innovation that was created to solve these problems. Among the three core elements of DevOps (People, Process, and Technology), the realization of technology, that is, CI/CD, has become essential.

In this article today, we will take a look at Flow, an innovative DevOps platform of Alibaba Cloud.

2. Solution Overview

Alibaba Cloud DevOps Flow (hereafter Flow) is a DevOps SaaS platform provided by Alibaba Cloud. It is an enterprise-level automated R&D and delivery pipeline platform, providing business with flexible and easy-to-use features of continuous integration, continuous verification, and continuous deployment. Open source projects that provide similar services include Jenkins, Bamboo, and ArgoCD.

Most of the traditional CI/CD tools use a specific script language and are difficult to configure. But with Flow, essentially all CI/CD Steps are provided in a GUI as shown in the capture below. You can easily access and use them without knowing a specific script language or specific usage.

3. Solution Function Description

3.1 Terminology

Source: Delivery tools such as Git repository
Phase: A collection of tasks executed in the pipeline, can be executed automatically or manually, and can be executed in two phases consecutively
Task: Actual action, two tasks can be executed simultaneously or consecutively. / Normal task includes code scanning, unit test, building, deployment, code merging, manual approval, etc.
Step: As the core basic function of Flow, actual operation is included.

3.2 Advantages

Easy Deployment: Create a workspace and use the pipeline right after a few minutes
Simple Management: Easy pipeline management with GUI
Tenant Isolation: Create multiple workspaces to isolate each other
High Stability and Reliability: Can be deployed in multiple zones
Various Supported Pipeline Sources: Integration with DSCM such as Github, Gitlab, etc.
Guaranteed Business Delivery with High Quality: Provides functions suitable for corporate environments such as code scanning, security scanning, and unit test

3.3 Features

3.3.1 Pipeline Source

Defines the Source Code Repository for starting the Pipeline. By default, GitHub and General Git are available.

GitHub can handle the credential for the repository by linking the user's account, but in General Git, you need to set the key, etc. in the options.

3.3.2 Tasks

Task defines the actions required in each CI/CD Step provided by Alibaba Cloud.

Alibaba Cloud pre-defines and provides various CI/CD tasks, and representative tasks will be explained below.

3.3.2.1 Java Code Scan

Code scanning for Java Source. Using the script provided by Alibaba Cloud, the high-risk code in each code is analyzed and a report is provided.

Task Name: Included in all tasks and define the subject of the task
Build Cluster: Defines the region that contains the cluster.
Step Name: Defines the subject of the step to be included in the task
JDK Version: Select JDK version
Maven Version: Select Maven Version
Enable Incremental Scan: Enable Incremental Scanning policy
Use Custom Rule: Use the p3c rule file in your code repository
Sub Folder: Used to determine the directory to be verified in the Repository, if it is blank, use the root directory
Excluded Folder: Defines the directory not to be included in Scanning.

After the pipeline runs, you can refer to the Code Scanning report. You can find out in detail which source code is the issue.

3.3.2.2 Maven Unit Test

Set options for unit test using Maven

JDK Version / Maven Version: Select the version of JDK and Maven
Test Command: Defines the command to be used for the test, and basically provides reporting using surefice
Test Report Folder: Define the folder to save the Test Report
Test Report Entry File: Define the test report entry file

3.3.2.3 Java Build

It is a task that builds Java source code, and Maven build is used in the default setting. If you use Java Build provided by Flow, you can easily build without using a separate build server when developing and deploying.

JDK Version: Defines the JDK version to be used for Java build.
Maven Version: Defines the Maven version
Build Command: Used to customize the command to be input when performing a build, define mvn options, etc.

3.3.2.4 Java Image Build

It is mainly used for container deployment. Usually, when deploying a container in a local environment, the developer pulls the Base Image from the registry and then performs Java build. After that, the built library and Base Image are merged to create an image to be operated.

If Java Image Build is used, users can easily create source code into container images without performing a separate build server or Docker build.

Java Build included in Configuration is the same as above.

First, define options for ACR build, which is Alibaba Cloud's Container Registry service for Docker build.

Choose Service Connection: Select a service to connect ACR service and credentials
Region: Select the region to use ACR
Repository: Select a repo of the previously distributed ACR, if it is not already created, you can create it with +
Dockerfile Path: Enter the Dockerfile Path to refer to when deploying the image.
Context Path: Enter the context path when entering Docker build command
No Cache: If checked, use the -no-cache=true option of docker build

3.3.2.5 ECS Deployment

It defines the ECS to be used when deploying. Depending on the configuration, it can be deployed to multiple ECSs.

Artifact: Defines Artifact Name
Deployment Group: Defines Deployment Group
Download Path: Defines the Artifact Path to be downloaded from the host, blank indicates no download
Execute User: Define the User to be executed in ECS
Deployment Script: Script to be used when deploying ECS/VM
Suspend Strategy: The choice of suspend method used in host groups such as Suspend First Batch, Suspend Per Batch or Without Suspension. Recommend Suspend First Batch
Batches: Defines the number of batches to divide the machines in the host group into.

3.3.2.6 Kubectl Apply

If you use Kubernetes for deployment, you can apply the Kubectl Apply task. You can also use a single container for deployment, but you can use Blue/Green or Canary deployments depending on your requirements.

Service Connection: Select Service Connection to be connected with Kubernetes to be deployed
Kubectl Version: Select the Kubernetes version to be deployed
Namespace: Defines the namespace where the container will be deployed.
YAML PATH: Defines the YAML file path to be used for deployment, mainly using the manifests file
Use Replace Mode: Use replace mode when creating target resource using 'kubectl install' instead of 'kubectl apply'
Skip TLS Validation: used when 'insecure-skip-tls-verify' is configured to true in Kubernetes configuration

3.3.2.7 Manual Approval

It is used when an approval procedure for a specific task is required during distribution. Basically, approval can be processed in the account registered in FLOW.

Validation Method: Select a group of users to approve (Anyone / All)
Validator Type: Select the type of approver (User / Tenant Role)

In addition, tasks such as Blue-green deployment, Node.js build / test, Execute command, and custom image repository build can be used.

3.3.3 Additional Features

FLOW's console provides a variety of additional functions for managing your pipeline.

3.3.3.1 WebHook Trigger

Users can tightly connect GIT repository and CI by using WebHook Trigger. You can use the API displayed in the menu easily by adding it to the GIT webhook.

3.3.3.2 Scheduled Trigger

A feature that automates the execution of pipelines, primarily for periodic tasks such as PMs.

3.3.3.3 Variable

Defines an Object to be used as a variable in the pipeline environment.

3.3.3.4 Cache

Caching is used to speed up pipeline deployment.

4. Test Background and Requirements

You can create a workspace with your Alibaba Cloud International account by accessing the Alibaba Cloud DevOps FLOW. Using this, you can easily configure the pipeline environment.

From item 5, you can check the guide for creating a pipeline that deploys ECS Single Instance using the source code stored in GitHub, assuming that there is a customer with the specific conditions below.

4.1 Background

The example environment has the following background.

Code development in Java
Delivery in the JAR or WAR format
Execution on ECS or a self-managed host

4.2 Requirements

The technical requirements for implementing the example environment are as follows.

Perform some quality checks on the source code such as unit testing and code scanning.
Build source code into a deliverable such as a JAR or WAR file.
Publish a deliverable to an ECS virtual host

5. Main Steps

In this example, the pipeline will consist of four stages.

You can check how to configure in the order of GitHub Repository Connection > Java Code Scan / Maven Unit Test > Java Build & Archive > ECS deployment.

Example source code can be forked at https://github.com/Jason-Jongjin-Lim/FLOWtest

5.1 FLOW Connection and Create Pipeline

If you connect to the FLOW Console and create a workspace, you can see the following screen. Here we choose Create Pipeline to configure the pipeline.

5.2 Template check and selection

If you select Create Template, you can check the list of predefined templates. We will set it up using the first Template here.

5.3 Add Pipeline Source

In this guide, we will use GitHub as the Pipeline Source. First, you can associate a GitHub account with FLOW and follow the guide below to set it up.

Namespace: Choose a Namespace to use on GitHub
Repository: Select Repository with source code to be distributed in Namespace
Default Branch: Select branch to deploy
Clone Submodule / Custom Clone Depth: Select False

5.4 Add Source Trigger

If the Enable Code Source Trigger option is turned on, the pipeline can be started automatically using the webhook function provided by the Git process. After setting this option, the developer executes the pipeline only with Commit/Push operation.

Enable the function and copy the Webhook API

Enter the copied API in GitHub Repository > Settings > Webhook > Payload URL

5.5 Java Code Scan Settings

Set up a scanning policy to check for potential issues in the sample Java code.

In this example, the default option is used. If there is a specially used JDK or Maven version, you can specify it.

5.6 Maven Unit Test Settings

Specify options for Maven Unit Test. As an example used in this scenario, the reporting function using surefire can be used.

In this example, the default option is used.

5.7 Build, Archive Task Settings

Specify each option for Java build.

For the rest, the default option is used and deploy.sh is added to the Archive Paths. This script is compressed into an archive when deployed and stored in ECS.

5.8 ECS Deployment Settings

Specifies the options to use when deploying with ECS.

You need a group of ECS to use for deployment. To do this, click Create Deployment Group.

Create Service Authorization in the pop-up window and select the region to deploy.
Select an instance to be used as a distribution server. Multiple instances can also be selected.

In the next step, designate the Group Environment and Tag to be used for the selected Server Group. In this environment, we will use the Daily Environment.

Decompress the package saved as archive in Deployment Script and plant the script that executes Deploy.sh.

mkdir -p /home/admin/application/
tar zxvf /home/admin/app/package.tgz -C /home/admin/application/
sh /home/admin/application/deploy.sh restart

5.9 Save and Run

When the pipeline creation is completed, execute Save and Run at the upper right of the FLOW Console.

5.10 Progress Confirmation

After running the pipeline, you can check various information about the pipeline progress in the Dashboard of the FLOW Console.

In the left panel, you can check basic information about the user who executed the pipeline, execution time, source, variable, etc.
Detailed information such as logs and reports for each step can be checked in the Pipeline panel

Click each step to check detailed information, process, and status

If you click Log in Java Build Step, you can check the log information of each build step.

When the pipeline ends, the status window shows all complete.

6. Troubleshooting

As the pipeline progresses, problems occur at various stages. If you check the detail page of the step where the problem occurred, you can troubleshoot such as detailed information, log, retry, and skip.

You can also check the status of previously executed pipelines in Running History.

7. Conclusion

In this blog, we have talked about Alibaba Cloud DevOps Flow, introducing its features and capabilities.

At the time of writing, Flow supports less plug-ins compared to Jenkins and Bamboo, both of which are currently widely used pipeline tools. Because of this, Flow may be a little light for corporate users who have complex DevOps environments.

However, it is suitable among Alibaba Cloud customers who operate simple services based on Java or Node.js. With Flow, customers can automate build, test, approval, and deployment to instances and containers with only Commit/Push after source code development, so it is a more suitable solution than heavy Pipeline tools.

Community