CI/CD framework based on Serverless architecture

Recently, Serverless Devs, a serverless developer platform, released a lightweight CI/CD framework based on Serverless architecture - Serverless-cd. Serverless-cd is a powerful, flexible, secure, and low-cost CI/CD open source framework running on the Serverless architecture. The framework is built on the basis of Serverless Devs developer tools, through which Serverless-cd developers can quickly build an enterprise internal application management PaaS platform.

Developers want to focus more on creating business value

Serverless Devs is a CNCF sandbox incubation project. It was open sourced by Alibaba Cloud in 2020. It is an open source and open Serverless developer platform. Create a one-stop serverless application development service for developers, help solve the current tool chain difficulties, let developers experience multi-cloud products with one click, and deploy serverless projects quickly.

The Serverless Devs project provides a full lifecycle solution for application development, debugging, deployment, operation and maintenance, and monitoring. In actual use, application deployment and delivery is still a difficult problem for developers. Developers want to focus more on application development and value creation—that is, on coding and building applications, rather than on application deployment and delivery.

Before the release of the Serverless-cd project, the Serverless Devs project integrated mainstream CI/CD tools through integration: https://github.com/Serverless-Devs/cicd, such as Jenkins, Github, Gitlab, etc., but using these Traditional CI/CD tools generally face the following problems:

1. Low utilization rate of resources: It is necessary to prepare the construction machine in advance, and when there is no construction task, the machine resources are wasted;
2. Task queuing: If the resource preparation is insufficient, the task queuing time is too long during the peak period of business construction;
3. Poor isolation: If a task consumes a large amount of computing/storage resources during execution, other tasks fail;
4. Security issues:
a. Use the CICD service provided by the platform: the code and the build machine are not in the same network environment, and public network access has to be opened, causing security issues.
b. ECS virtual machine deployment: multiple applications are built on one instance at the same time, and a malicious application can access the code of other applications

2 characteristics of CI/CD pipeline

CI/CD pipelines have two notable features:
1. Event-driven
Whether it is automatically triggered by receiving Webhook or manually triggered by calling Open Api, for the CICD system, it is passively receiving instructions for consumption
2. Obvious peaks and troughs in business
The peaks that trigger CI/CD builds are generally during working hours, and there are fewer build tasks after get off work and at night. At the same time, some tasks are very time-consuming to execute, and some tasks require a large amount of CPU and memory resources, so it is difficult to perform effective capacity estimation in advance.
• Too little preparation of machine resources: Task execution fails due to insufficient resources, or multiple tasks are preempting resources and cannot be executed.
• Excessive preparation of machine resources: cannot be fully utilized, resulting in waste of idle resources.

CI/CD Advantages Based on Serverless Architecture

Under the Serverless architecture, CI/CD can have the following advantages

1. Automatic Elasticity
The serverless platform will allocate a brand new instance for each build task to ensure that each task does not affect each other. You no longer have to worry about task failure due to insufficient resources, and avoid the situation that tasks cannot be queued due to insufficient resources.
2. Pay for value
During the business trough (evening or after get off work), there are only a few or no tasks to be executed, and resources are idle and wasted. The idea of serverless is to help customers pay for the actual value generated. Only when the construction behavior actually occurs, will the cost be incurred.
3. Free operation and maintenance
Serverless elasticity is horizontal expansion according to the request. Developers do not need to pay attention to the underlying resource scheduling and operation and maintenance work, and can realize business development and value creation without distraction.

Serverless-cd technical architecture

Serverless-cd fully follows the best practices of serverless architecture and refers to the implementation of Github Action at the specification and ecological level. The following is a piece of YAML for Serverless-cd to deploy the Serverless Devs application:
name: "Deploy Express application to FC"
steps:
- run: npm i @serverless-devs/s -g --registry=https://registry.npmmirror.com
- run: s -v
- run: echo ${{secrets.ALIYUN_ACCOUNTID}}
- run: echo ${{secrets.ALIYUN_AK}}
- run: echo ${{secrets.ALIYUN_SK}}
- run: s config add --AccountID ${{secrets.ALIYUN_ACCOUNTID}} --AccessKeyID ${{secrets.SIMPLE_ALIYUN_AK}} --AccessKeySecret ${{secrets.SIMPLE_ALIYUN_SK}} -a default -f
- run: s deploy --use-local -y

Serverless-cd adopts the classic Master Worker model and an event-driven architecture. The overall architecture is as follows:

trigger method

Triggers assume the role of producers in event-driven, and Serveless-cd temporarily provides three trigger methods:

1. Automatically triggered by Webhook:
Developers can configure corresponding trigger conditions: for example, Push to the Master branch and initiate a Merge Request. This is a very agile way of developing and delivering
2. Triggered by Open API:
All capabilities of the Serverless CD platform provide open APIs. So that developers can better build the PaaS platform inside the enterprise
3. Trigger via CLI:
From a technical point of view, Serverless-Devs is essentially a CLI job that provides componentization capabilities. The so-called componentization is to allow developers to expand according to their own needs through the hot update mechanism. Serverless-cd allows developers to directly operate through the command line through custom components.
Of course, we are also planning to access more trigger types, such as cloudevents trigger, timing trigger, etc.

Serverless (FaaS) platform

The FaaS platform is the core part of the entire platform, taking on the role of consumers in event-driven, using the classic Master Worker model.

👉Function Compute FC: https://www.aliyun.com/product/fc?

Advantages of Master Worker Model in Serverless Architecture

In the traditional Master Woker deployment, in order to ensure the high availability of the Master node, it is necessary to deploy three nodes with load balancing and health check to ensure the high availability of the Master node. In the serverless architecture, the instance will automatically expand elastically according to the request, and the master node naturally has high availability, which has great advantages in terms of reliability and flexibility

Master function

The Master function is essentially an HTTP type function. As the entrance of the overall traffic, it is also the brain of the entire system and bears very important responsibilities.

1. Security
• Public network key verification
The URL address exposed by the Master can be accessed from the public network. In order to prevent malicious requests, the serverless-cd system will also issue signature rules when issuing URLs. If it is a malicious request, it cannot pass the verification to ensure the security of the system
• VPC binding
Binding to a VPC environment is also supported. The code repository and serverless-cd service are bound to the same VPC environment and triggered through a Webhook. Public network users cannot directly access, ensuring absolute security from the network
2. Filter requests
We trigger various events in Git Repository every day, such as creating new ISSUE, PUSH code, launching Merge Request, etc. These actions will be triggered through Webhook, and developers can configure rules to filter related events. The following is an example: only the code submitted to Master on the GitHub platform will trigger
triggers:
github:
events:
- eventName: "push"
filter: 'body.ref in ["refs/heads/master"]'
3. Route forwarding
The Master function is responsible for distributing the request to the Worker function, and all the behaviors of the Worker function are controlled by the Master function

Worker function role

The Worker function is essentially an event function, which only communicates with the Master function, and its only responsibility is to process the Pipeline, which can run for a long time.

custom pipeline
serverless-cd supports three ways to customize the pipeline
shell script
Shell scripts are the easiest to understand and most widely used
name: "shell example"
steps:
- run: echo Hello world
zx script
google/zx allows developers to write your shell scripts through javascript syntax
name: "zx example"
steps:
- script: 'const listFile = await $`ls -la`; console.log(listFile)',
Extend with a custom application (NPM Package)
Serverless-cd also supports encapsulation of common NPM packages for expansion: general capabilities such as DingTalk notifications, corporate WeChat notifications, and OSS file uploads can all be extended through custom applications. The essence of a custom application is a package published in the NPM repository.
name: "zx example"
steps:
- run: @serverless-cd/dingding
Quick experience
Prepare in advance
The deployment of serverless-cd is completely based on the cloud, and relying on related cloud products is also serverless.
• Function Compute FC: The computing power of the entire system runs on FC
• Log Service SLS: Distributed log storage service to better locate and discover problems
• Object storage OSS: used to store log information
• Tablestore Tablestore: cloud data information storage for applications and tasks

local deployment

1. Download and install Serverless Devs: npm install @serverless-devs -g (version must be greater than 2.1.7), please refer to the Serverless Devs installation documentation for detailed instructions
2. Configure key information: s config add, for detailed operation guidance, please refer to Configuring Alibaba Cloud Key
3. Initialize the project: s init serverless-cd
4. Enter the project and deploy: cd serverless-cd && s deploy
Serverless-cd RoadMap

Open source co-construction
Serverless-cd is the industry's first exploration of CI/CD based on serverless architecture. Welcome to our open source address: https://github.com/Serverless-Devs/serverless-cd. Serverless-cd has just been open-sourced, and there are still a lot of details and work to be done. We look forward to working with more developers to build the Serevrless ecosystem, so that developers can focus on business development and value creation without distraction.