×
Community Blog Leverage your CI/CD pipeline by Autoscaling Jenkins Agent with Alibaba Cloud

Leverage your CI/CD pipeline by Autoscaling Jenkins Agent with Alibaba Cloud

Jenkins is tools that widely used for automation server, some use cases of Jenkins is for implementing a CI/CD pipeline for automate applications depl...

By Rifandy Zulvan, Solution Architect Intern

Introduction

Jenkins is tools that widely used for automation server, some use cases of Jenkins is for implementing a CI/CD pipeline for automate applications deployment pipelines like installing dependencies, build, test, and deploy your applications.

Autoscaling your Jenkins Agent can leverage your CI/CD pipeline to handle more pipelines without having to worry about high build queues. This solution can be done with Alibaba Cloud by using Alibaba Cloud ECS plugin on Jenkins, this also can be a cost-effective solution for your CI/CD pipeline by using Ephemeral ECS instance as your Jenkins Agents.

Jenkins Overview

The common architecture for CI/CD in Jenkins is Master-Slave architecture. In this architecture, we have one Jenkins Master that for controlling and manage some resources, configurations on the Jenkins. And we can have lot of slaves as an agent to do some Jenkins Jobs.

image_jpeg

This architecture can be used for a highly load build, or for a different use cases build (Android App Build, IOS App Build, Docker container build, etc). This method can speed up you DevOps implementation, because with a high load of build it can be handled with multiple of slaves, on this blog will gives you a guide to Auto Scale your Jenkins Slaves using Alibaba Cloud, so it can be more effective to handle a lot of build.

Jenkins ECS Plugin

Because Jenkins is an open source tools, and it has a quite big communities in Open Source industry. Jenkins provides a lot of plugin and to support your automation needs. One of the Plugins is Alibaba Cloud ECS Plugin.

Alibaba Cloud ECS Plugin can be used to implement an auto provisioning you Jenkins Slave node. This is the mechanism of the Alibaba Cloud ECS Plugin.

image

So basically, the plugin can provision Alibaba Cloud ECS instances that can be used automatically as Jenkins Slave. With this plugin, you don’t need to set a manual configuration to assign each nodes IP addresses on your Jenkins Master.

This plugin can also be used to automatically scale up your slave, base on slave availability. This flowchart shows the mechanism of how this plugin can scale up your slaves.

image_3

One more important feature on this plugin is that you can use Ephemeral ECS instances and On-Demand ECS instances. Using Ephemeral ECS instances as your Jenkins Slave is a cost-effective solution because the Jenkins slaves do not need to be available at all times. Instead, they can be provisioned only when needed and deleted if the slaves are in idle state.

This is a price comparison of on-demand and ephemeral instances on Alibaba Cloud. For further information you can visit the ECS documentation Overview (alibabacloud.com).

image_4

Installation

This part will show you how to install the Alibaba Cloud ECS Plugin on your Jenkins Master. There are some prerequisites that you need to provide.

  1. Jenkins Master (on-premises, on-cloud)
  2. Alibaba Cloud Account Access Key
  3. SSH Key pair
  4. VPC and VSwitch
  5. ECS Image Snapshot (optional)

I assume you already have a Jenkins Master on your side, if you haven’t you can look at this Alibaba Cloud’s blog about Continuous Integration with Jenkins on Alibaba Cloud.

Step 1 – Alibaba Cloud Account AccessKey

The first thing to provide is an Account Access Key, you can get it on Resource Access Management (aliyun.com) page on Alibaba Cloud Console, and just click Create AccessKey

image_5

After that you can Copy or Download the AccessKey as CSV file, we need to use this Key to set up the plugin later.

image_6

Step 2 – Create SSH Key Pair

To create an SSH Key Pair, you can go to ECS console and then Navigate to Network and Security > Key Pair menu and Click Create SSH Key Pair.

image_7

After that, you can set your SSH key pair name, creation type, resource group and tag.

image_8

And then you can click OK and you will get the key file with ‘.pem’ format, keep this file we will use the key later.

image_9

After this, we can start to install the Alibaba Cloud ECS Plugin on Jenkins Master.

Step 3 – Install Alibaba Cloud ECS Plugin on Jenkins Master

Go to Jenkins Master, and go to Manage Jenkins > Manage Plugins.
Next step, go to Available plugins and search Alibaba Cloud on the search box, click install on the Alibaba Cloud ECS plugin, and then click Install without restart.

image_10

image_10

image_11

Wait for the plugin to installed, and the Plugin is already installed on your Jenkins Master.
To check the plugin has been installed, you can go to Plugin Manager > Installed plugins > and check the Alibaba Cloud ECS plugin.

Step 4 – Configures Alibaba Cloud ECS Plugin

To configure Alibaba Cloud ECS as your Jenkins Slave, you can go back to Jenkins homepage, and go to Build Executor Status > Configure Clouds.

Annotation_2023_05_29_102623

And on the Configure Clouds page, you can go click Add a new cloud and choose Alibaba Cloud ECS.

image_12

Set the name of your Alibaba cloud ECS. If your Jenkins Master is running on Alibaba Cloud and should be running at same VPC with your Slave, Check the Jenkins Master in Private Network options, this option will run and connect the provision ECS Instance with the master with private IP address. If you have a multi-cloud or hybrid architecture you can uncheck this option, For this Blog we uncheck this option.

Annotation_2023_05_29_102912

Step 4.1 – Configures Alibaba Cloud Credentials

Next, we need to provide Jenkins our Alibaba Cloud AccessKey. Go to Alibaba Cloud Credentials and click “Add” > “Jenkins”

image_14

Choose the credentials kind as Alibaba Cloud Credentials, add your Access Key ID and Secret Access Key, and add the ID and description also, after that you can click Add.

image_15

Next, add your credentials, region, VPC, and Security Group.

image_16

The next step is to add your ECS SSH Key, click Add > Jenkins and enter your ECS username and your private key.

image_17

image_18

After entering your SSH username and private key, click Advanced on the configuration and check Assign Public IP and No delay provision settings. You can configure this advanced configuration as your need, if your Jenkins Master running on the same VPC with your Jenkins Slave, you don’t need to Assign Public IP configurations.

After that you can do a connection test to Alibaba Cloud from your Jenkins Master, you can click Test Connection. If there is a connection OK message on the left, your configurations are already correct.

image_19

Step 4.2 – Configures Slave Image

To run a slave you need to configures your Slave Images, the best practice is set an image that can support your Jenkins Jobs task. For example, if you are running a CI/CD pipeline that compile a Golang Application, you need to select an image that already have a Golang installed.
You can create your custom image, or you can choose Alibaba Cloud existing images, for this blog we only choose an existing image.

To start configuring an image, you can click Add on Images configuration.

image_20

And then, you can add your image description, and set the Image. If you are using custom Image, you can insert the ID of your custom image, If you are using an existing image, you can choose any images as your demand on this options.

Annotation_2023_05_29_103751

Next, you can specify your Availability zone, Vswitch, Instance Charge Type,and Instance Type.

Annotation_2023_05_29_104133

And you can configure your slave’s root directory, disk type and size.

Annotation_2023_05_29_104432

The next step is to configure minimum and maximum instances number, agent label, instance name. After that, you can test the ECS creation by clicking the TestCreateECS, if there is a Success message, you can save the configuration and your ECS Slave configurations are ready.

image_23

Annotation_2023_05_29_104800

5 - Create Jenkins Pipeline

To test the slave auto-scaling, we need to create Jenkins Pipeline for our Slave to execute. In this blog we just use sample pipeline like here. To do the slave auto-scale, keep in mind to assign agent pipeline label to your ECS plugin label.

pipeline {
    agent {
        label 'ecs'
    }
    stages {
        stage('test') {
            steps {
                echo 'hello'
                sleep 60
            }
        }
    }
}

Here's how slave autoscaling works in this scenario, where there are no ECS slaves to begin with, when the build begins, the pipeline will be on the Build Queue, and the ECS slave will provision for roughly 2-5 minutes.

lALPM4AfHKajcy3NAa7NB2I_1890_430

After the ECS slave provisioned the build is automatically assign to the existing slave.

Annotation_2023_05_29_111156

Conclusion

In DevOps practices, CI/CD is a crucial tool. Alibaba Cloud ECS instance can be used as part of your CI/CD tools with Jenkins to automatically scale up your Jenkins Agents. You can also create a cost-effective CI/CD pipeline by using Ephemeral ECS instances as your Jenkins Master. This approach allows you to leverage the power and flexibility of Alibaba Cloud ECS on your Jenkins CI/CD pipeline.

0 1 0
Share on

Alibaba Cloud Indonesia

94 posts | 12 followers

You may also like

Comments