By Rick Yang
Nowadays, DevOps, IaC, and CI/CD have been discussed and adopted in different ways. It's difficult to tell which one is better than the other. However, only you and your team know which one is suitable for your specific situation. With that said, as a Solutions Architect from Alibaba Cloud, I decided to put my feet into my customer's shoes and made a CI/CD demo with open-source and Alibaba Cloud managed services.
Again, it may not be the best practice, but please treat it as a starting point for further discussion. Let's get started with the following objectives.
Objectives:
Create a CI/CD pipeline consisting of three main stages: Test, Build/Push, Deploy
This web application is a demo API endpoint that can be accessed from a local host, as illustrated below:
Build the Docker image of this web application from scratch on your PC/Mac/Linux host:
mkdir fastapi_app
requirements.txt
file with the following content:fastapi>=0.68.0,<0.69.0
pydantic>=1.8.0,<2.0.0
uvicorn>=0.15.0,<0.16.0
fastapi_app
and name it app
app
folder: an empty file __init__.py
and an main.py
with the following content:from typing import Union
from fastapi import FastAPI
app = FastAPI()
@app.get("/")
def read_root():
return {"Hello": "World"}
@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
return {"item_id": item_id, "q": q}
#
FROM python:3.9
#
WORKDIR /code
#
COPY ./requirements.txt /code/requirements.txt
#
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
#
COPY ./app /code/app
#
CMD ["uvicorn", "app.main:app", "--proxy-headers", "--host", "0.0.0.0", "--port", "80"]
.
├── app
│ ├── __init__.py
│ └── main.py
├── Dockerfile
└── requirements.txt
requirements.txt
file and execute the following commands (make sure you have installed Docker on your host. If not, please check this link)docker build -t myimage
docker run -d --name mycontainer -p 80:80 myimage
http://127.0.0.1
Now, we can make sure the web application can be built and run locally without issues. Next, let's upload our code to the GitLab repository.
There are two ways to get this done, signing up for a gitlab.com account or creating a self-managed gitlab.yourdomain.com. I chose the second one, but either way is fine for this demo.
Create the GitLab project and upload the code files to it:
Before creating the CI/CD pipeline on GitLab, we need to set up a k8s cluster for the web application deployment. Alibaba Serverless Kubernetes or ASK is used for this demo because I am from Alibaba Cloud :smile:, and it has the following features to help DevOps teams easily plan, test, and deploy their applications in an on-demand, fast, reliable, cost-effective, and secured way. Please click this link for more details about ASK
Terraform is a well-known Infrastructure as Code (IaC) tool developed by HashiCorp.
With Providers and Modules, Terraform is pre-integrated with different IT vendors and cloud service providers, including Alibaba Cloud. Please click here for the latest documentation.
The terraform-cli need to be installed on your local host as a precondition to the following steps. You should have an Alibaba Cloud RAM account with ACCESS_KEY and SECRET_KEY to access the Alibaba Cloud backend from Terraform.
Please check this link for more information
ask_demo
to host the Terraform filesmain.tf
, terraform.tfvars
, and variables.tf main.tf
contents:terraform {
required_providers {
alicloud = {
source = "aliyun/alicloud"
version = "1.174.0"
}
}
}
//1) setup of alicloud RAM account with resource permisions and obtain the access_key and access_key_secret
provider "alicloud" {
access_key = var.my_access_key
secret_key = var.my_secret_key
region = var.my_region
}
data "alicloud_zones" "abc_zones" {}
//2) setup of VPC
resource "alicloud_vpc" "test_vpc" {
vpc_name = var.my_vpc
cidr_block = var.my_vpc_cidr_block
}
//3) setup of vswitch within the VPC
resource "alicloud_vswitch" "test_vswitch" {
vswitch_name = var.my_vswitch
vpc_id = alicloud_vpc.test_vpc.id
cidr_block = var.my_vswitch_cidr_block
zone_id = data.alicloud_zones.abc_zones.zones.0.id
}
resource "alicloud_vswitch" "test_vswitch_bak" {
vswitch_name = var.my_vswitch_bak
vpc_id = alicloud_vpc.test_vpc.id
cidr_block = var.my_vswitch_cidr_block_bak
zone_id = data.alicloud_zones.abc_zones.zones.2.id
}
resource "alicloud_cs_serverless_kubernetes" "serverless" {
name = var.my_ask_name
version = "1.22.3-aliyun.1"
vpc_id = alicloud_vpc.test_vpc.id
vswitch_ids = [alicloud_vswitch.test_vswitch.id,alicloud_vswitch.test_vswitch_bak.id]
new_nat_gateway = true
endpoint_public_access_enabled = true
deletion_protection = false
load_balancer_spec = "slb.s2.small"
time_zone = "Asia/Shanghai"
service_cidr = "172.21.0.0/20"
service_discovery_types = ["PrivateZone"]
}
terraform.tfvars
contents
Replace "YOUR_ACCESS_KEY" and "YOUR_SECRET_KEY" with your RAM account information:
my_access_key = "YOUR_ACCESS_KEY"
my_secret_key = "YOUR_SECRET_KEY"
variables.tf
contents
variable "my_access_key" {
description = "RAM user access_key"
sensitive = true
}
variable "my_secret_key" {
description = "RAM user secret_key"
sensitive = true
}
variable "my_region" {
default = "cn-hongkong"
}
variable "my_vpc" {
default = "test_vpc"
}
variable "my_vpc_cidr_block" {
default = "10.1.0.0/21"
}
variable "my_vswitch" {
default = "test_vswitch"
}
variable "my_vswitch_bak" {
default = "test_vswitch_bak"
}
variable "my_vswitch_cidr_block" {
default = "10.1.0.0/24"
}
variable "my_vswitch_cidr_block_bak" {
default = "10.1.1.0/24"
}
variable "my_ask_name" {
default = "ask001"
}
terraform init
: If no error has been thrown out, you should see something like this:terraform plan
command to prepare all the resources to be deployed on Alibaba Cloud without really creating them at this stage:terraform apply
command with your confirmation to deploy the planned resources on Alibaba Cloud. It may take a few minutes to create the ASK cluster for this case.Until now, we have GitLab for source code management and CI/CD pipeline. We also created the ASK k8s cluster for the application deployment, but how about the Docker image repository for image PUSH and PULL? We should have a place to store and manage the Docker image. This is where ACR comes into play.
Compared to the previous steps, creating the ACR is easy.
You can find the login and pull and push instructions within the detail information page of the repository you just created, which will be used in your CI/CD pipeline configuration:
We need to have the yaml files for Deployment, Service, and Ingress creation to use the kubectl CLI tool to deploy the k8s based application. There are two ways to get there. You can create the files from scratch or copy them from the ASK GUI Console. I prefer the second option, which can save a lot of effort.
To generate the yaml files of Deployment, Service, and Ingress, we need to create them manually with the ASK GUI Console:
Enter the name and the number of replicas (pods) and click Next:
Input the name and tag of the image, which can be copied from the ACR page in the previous step. We also need to specify the username and password we created for our ACR by clicking the Set Image Pull Secret and inputting the credentials as requested.
We should see the pod is up and running with the Docker image pulling from ACR:
After the yaml files are ready, we can complete the last step of this demo, the CI/CD pipeline.
Let's go back to the GitLab page and navigate to the project we created beforehand. Add a new file and name it .gitlab-ci.yml
, which is used for editing the CI/CD scripts.
Click Edit and input the following contents with three stages: test, build, and deploy. As the names suggest:
The test stage is running the unit test script (Please find the test.py in the appendix section).
The build stage is to build the Docker image from the code and push it to the ACR.
The $ACR_USER and $ACR_PASSWORD represent the username and password created in ACR for image pushing authentication. They can be configured in the Settings—CI/CD—Variables.
The deploy stage is – First, login into a jumper host or local host and execute the kubectl to deploy the web application to the ACK k8s cluster. Assume the jumper host or local host has been installed with the kubectl CLI tool and configured with the $HOMEDIR/.kube/config
file with ACK's connection information:
stages:
- test
- build
- deploy
run_test:
stage: test
image: python:3.9-slim-buster
script:
- python3 test.py
build_image:
stage: build
image: docker:20.10.16
services:
- docker:20.10.16-dind
variables:
DOCKER_TLS_CERTDIR: "/certs"
script:
- docker login -u $ACR_USER -p $ACR_PASSWORD registry-intl.cn-hongkong.aliyuncs.com
- docker build -t registry-intl.cn-hongkong.aliyuncs.com/rickyang_namespace/rickyang_repo:myapp-1.0 .
- docker push registry-intl.cn-hongkong.aliyuncs.com/rickyang_namespace/rickyang_repo:myapp-1.0
deploy:
stage: deploy
script:
- chmod 400 $SSH_KEY
- ssh -o StrictHostKeyChecking=no -i $SSH_KEY root@8.219.190.199 "
./test.sh &&
kubectl create -f testapp.yml &&
kubectl create -f testapp-svc.yml &&
kubectl create -f testapp-ingress.yml &&
touch mark.txt"
Finally, we can test our CI/CD pipeline. We need to update our code and click Commit Changes to trigger the pipeline. Then, GitLab, ACK, and ACR will take care of the rest.
Please remember to stop and remove the resources on ASK to save costs. It's super easy to do that with Terraform!
Open the terminal where you have your terraform-cli installed and go to the folder storing your three terraform files. Execute terraform destroy
with confirmation:
Alright, we have completed a demonstration of a CI/CD pipeline consisting of Terraform, GitLab, ACR, and ACK. You must have preferences on each step in real-life scenarios. Then, try it and improve it to adapt to the dynamic business needs.
It is a one-click deployment solution, even though we took a lot of effort to make it happen. If this task can be repeated hundreds of times, isn't it worthy?
The test.py
file is a mock file used to simulate the unit test.
import unittest
class TestSum(unittest.TestCase):
def test_sum(self):
self.assertEqual(sum([1, 2, 3]), 6, "Should be 6")
def test_sum_tuple(self):
self.assertEqual(sum((1, 2, 3)), 6, "Should be 6")
if __name__ == '__main__':
unittest.main()
The test.sh
file in the jumper host is used to detect if the application has been deployed. If yes, the existing one will be deleted and re-deployed again.
#!/bin/bash
if [ -f "mark.txt" ];
then
kubectl delete -f testapp-ingress.yml
kubectl delete -f testapp-svc.yml
kubectl delete -f testapp.yml
rm mark.txt
else
echo "The testapp does not exist yet"
fi
Alibaba Cloud's Carbon Emission Analyzing Tool for Businesses
1,071 posts | 263 followers
FollowHaemi Kim - October 20, 2021
Alibaba Cloud Community - August 3, 2022
Alibaba Cloud Community - July 22, 2022
JDP - July 31, 2020
JJ Lim - September 17, 2021
JJ Lim - September 23, 2021
1,071 posts | 263 followers
FollowAccelerate software development and delivery by integrating DevOps with the cloud
Learn MoreAn enterprise-level continuous delivery tool.
Learn MoreAlibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn MoreHigh Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.
Learn MoreMore Posts by Alibaba Cloud Community