These days, having a continuous integration and delivery (CI/CD) pipeline in place is pretty much essential for any kind of software development. Therefore, the question isn’t so much if you should set up a CI/CD pipeline, but how.
And when it comes to the how of CI/CD, you have a lot of options. There are countless CI/CD solutions on the market, including commercial as well as open source platforms, both of which have their own distinct advantages and disadvantages over the other.
In this article, I outline the reasons why you may wish to use a commercial CI/CD platform, then walk through the steps of taking advantage of tools on Alibaba Cloud that allow you to set up a production-ready CI/CD pipeline.
A commercial CI/CD platform is typically a hosted service that deals with the build and deployment steps in a standard pipeline. While there are privacy concerns that can be raised with commercial platforms, and the cost is generally higher than self-hosted solutions, they are a great way to establish a CI/CD pipeline with minimal resources. In addition to hosted services, on-premises commercial CI/CD solutions are available as well, which can help get around the privacy and security concerns that exist with hosted services.
Many commercial CI/CD platforms give back to the developer community by open-sourcing their CI/CD engines. While not every commercial platform open-sources their engine, and not every open source engine is backed by a commercial platform, these solutions provide a lot more control at the expense of ease-of-use. Spinning up an open source CI/CD platform is significantly more involved, and requires some sort of server to run on. Open source platforms are an excellent option for organizations that require more resources for their build processes, or even organizations that simply prefer to keep their code in-house whenever possible.
When it comes to hosting an application on Alibaba Cloud, both CI/CD solutions are viable—depending on the needs and use-cases of the application. To demonstrate, let's take a look at how to accomplish running remote commands and deploying a simple application using either a commercial or open source CI/CD platform. While the configuration of open source and commercial solutions differ, the process of actually implementing a CI/CD pipeline is pretty similar across the board, and interacting with servers—both in a private and public cloud—comes down to two commands: rsync and ssh.
Deployments in a Nutshell
Before we get started, let's assume that our CI/CD pipeline consists of only three parts: test -> build -> deploy. While the test and build processes can happen directly within the platform, deploying a build means that we will have to get creative with our deployment method. Some cloud providers offer APIs that make this solution feel more streamlined, but at a high level, the following method will work with any VPS. Thankfully, we can deploy any built code to a running ECS instance using standard shell commands:
rsync is a command line utility that is used to synchronize files across servers. It has been around for a while—The rsync algorithm is well-known as having been used by Dropbox to deal with file synchronization. The beauty of this deployment method is that it can be used regardless of the hosting provider you use, which means that you can deploy to multiple cloud providers simultaneously, both internally at Alibaba Cloud, and externally.
It is important to note that, in order to successfully authenticate to our ECS instance, you will have to generate an SSH key and give your CI/CD platform access to it. While out of the scope of this article, Digital Ocean has an excellent guide for generating SSH keys. (Take note: If you follow the Digital Ocean guide, you should not set up an SSH passphrase—otherwise your automated deploy process won't be able to use your key.)
Learn the following content, you can refer to Setting up a Multi-Cloud CI/CD Chain using Alibaba Tools.
This article is based on the speech delivered by Hua Xiang at "Kubernetes & Cloud Native Meetup - Guangzhou". It introduces some tools and steps for Alibaba to build a CI/CD pipeline.
Hello everyone, I am Hua Xiang from the Alibaba Cloud Container Service Team. First, let me explain briefly what Kubernetes is, to help you understand. Kubernetes is a container orchestration system available for production. Kubernetes makes all Node resources a resource pool in the cluster, and its scheduling unit is the Pod. A Pod can have multiple containers. It is just like a person holding ECS resources or computing resources in their left hand, holding containers in their right hand, and then matches the two, and in this way the person is playing the role of a container orchestration system.
However, the Cloud Native concept comes up quite frequently lately, and many people are confused about the connection between Cloud Native and Kubernetes. How can we determine whether an application is a Cloud Native application? In my opinion, three criteria are available:
First, the application can make resources a pool;
Second, the application can quickly access the network of the pool. Kubernetes has a layer of its own independent network, and I only need to specify the service name I want to access, that is, it can quickly access various service discovery functions through the service mesh;
Third, the application has the failover function. If a pool contains a host, or a node is down, and thus the entire application is unavailable, then it is definitely not a Cloud Native application.
From these three points, we can see that Kubernetes is doing very well. First, let's look at the concept of a resource pool. A large Kubernetes cluster is a resource pool. We no longer have to worry about the host of an application. All we have to do is publish the deployed yaml file to Kubernetes. It will automatically make these schedules, and it can quickly access the network of the entire application. In addition, the failover is also automatic. Next, I will share with you how to implement an elastic CI/CD system based on Kubernetes.
First, let's take a look at the current status of CI/CD. The concept of CI/CD has been put forward for many years. However, with the evolution of technology and the continuous introduction of new tools, the entire process and implementation methods of CI/CD are gradually enriched. Initially, we typically use CI/CD to submit code, trigger an event, and then perform automatic builds on a CI/CD system.
It involves many processes. The pipeline starts with triggering an event after the code submission. Then, the CI/CD system can build a layer "Build" through Maven, perform a unit test, perform code specification scanning, and then deploy the service. Next, it performs an end-to-end test for UI, which automatically tests the UI.
Then, it performs a stress test (a performance stress test), which is only done at the development and testing environment levels. Then, it can progress to a QA environment, and eventually to a UAT environment. The pipeline is a very long process. CI/CD is widely used. The code writing and submission to the code repository can be taken as the starting point of the entire IT infrastructure. From this point on, every step can be included in the scope of CI/CD. However, as the scope of its control is increasing, the entire process becomes increasingly complex, and it also occupies more resources.
If you know about C++, then you may know that C++ used to be a well-known language with a very long build time. One of the authors of the Go language, who is also an author of the C language, talked about writing the Go language simply as an alternative to writing code that took so long to compile. Therefore, one of the major advantages of the Go language is that the compilation time is very short. The compilation time of Go is indeed very short.
I used an I5 notebook to compile the Kubernetes source code for a complete build. It took about 45 minutes to compile, which was a very long process. Therefore, even if the compilation process has been greatly optimized and improved, as long as the project is large, the build phase would be long, not to mention some subsequent automated tests. Furthermore, the CI/CD process consumes and occupies a very large amount of resources.
This article shares how to implement a distributed and dockerized Jenkins Slave Pool on Alibaba Cloud which enables dynamic and parallel CI/CD pipeline provisioning to accomplish best resource utilization, on-demand scalability and cost-effectiveness.
Hybrid CI/CD is a distributed platform that uses a mix of on-premises and public cloud elastic compute resources for CI/CD purpose. Hybrid CI/CD basically control the workloads between private and public clouds to balance needs and costs.
At a high level, the architecture is shown in below diagram.
On-premises do offer a certain level of scalability, which is still per data center and highly depends on their budget. And often times when one system consumes resources than expected, all other systems are affected. So how to set the boundary for budgeting becomes tricky. Public cloud services will offer greater scalability with larger cloud infrastructure. Considering CI/CD jobs are usually isolated, short lived and generally stateless. By moving CI/CD jobs to the public cloud, it allows an organization to reserve scalability for the core services.
Public clouds are more likely to offer more economic cost effectiveness as the cost of centralized management are shared by every user.
Compared to pure public cloud solution, Hybrid CI/CD reserves the sensitiveness for security demand as you can always restrict data to be accessible only to applicable CI/CD jobs.
In the previous tutorials of this three-part tutorial series, we looked at installing Jenkins on Ubuntu 16.04 and created a sample Java application with Maven. We also created a build job in Jenkins to build the sample web application and successfully executed our first build job.
In this last tutorial of the series, we will configure Jenkins for automating the build job. We will also look at the basics of continuous delivery by configuring Jenkins to automatically deliver our web application to a Tomcat web server.
From your Jenkins dashboard, click on the sample job "MyMavenApp" to open the status page of the job. Now, click on the "Configure" link to open the configuration for the build job. Scroll down to find the "Build Triggers" section. You will see that the section has multiple options for the build triggers. For now, we are interested only in "Poll SCM" option. Select the checkbox and a text box to enter the polling schedule will appear. Schedule in Jenkins is set using slightly modified cron syntax.
In the above screenshot, I have used H/5 which will poll for the changes in the source in every five minutes. In a production environment, you may want to poll the changes for a less frequent duration, such as in every 6 hours. You can use H /6 to achieve this. Once you have provided the cron schedule, save the configuration. You will see a new option has been added to the sidebar with the name "Git Polling Log". You can use this option to learn about the last git polling.
To check, if Jenkins is triggering an auto build if there is some change in source code, we will make some change in the code. If everything is working than Jenkins should detect the change in next git poll. Once the change is detected, the automatic build should be started by Jenkins.
Log in to the terminal of your ECS instance using the sudo user. Switch to the directory where we have created the Maven sample project.
This course introduces the basic concepts, features and benifits of CI/CD compared with traditional software developing mode. Then we have a detailed introduction about the features and benefits of Jenkins software and also the installation of Jenkins based on Ubuntu operating system. Finally we introduced the detailed operations on Jenkins console with a brief demonstration.
In efforts to improve your e-commerce or online business, you will not only learn the fundamental philosophies of DevOps workflow, but also learn about the processes and tools needed to release software faster and more reliably, while obtaining industry standard certificates boosting your professional cloud credibility.
This course introduces the basic concepts, features and benifits of CI/CD compared with traditional software developing mode. Then we have a detailed introduction about the features and benefits of Jenkins software and also the installation of Jenkins based on Ubuntu operating system. Finally we introduced the detailed operations on Jenkins console with a brief demonstration.
Jenkins is an automation server with an unparalleled plugin ecosystem for supporting practically every tool as a part of the delivery pipelines.
Until now we have been focusing on adding new functionalities to our application (HTTPS and centralized logs). However, in doing so we have slowed down substantially our CI/CD pipeline, as it now takes about one hour to complete the full process.
The goal of this tutorial part is to focus on this slow pipeline problem and to find ways to accelerate it.
The slowest stage of our pipeline is the one responsible for deployment, and its first task is to download and install tools. It usually takes several minutes to complete and unnecessarily wastes resources such as network bandwidth.
A way to speed up this first task is to create our own Docker image, and then use it in our pipeline.
Concourse CI, a CI/CD tool whose charm lies in the minimalist design, is widely applied to the CI/CD of each Cloud Foundry module. Concourse CI officially provides the standard Docker images and you can use Alibaba Cloud Container Service to deploy a set of Concourse CI applications rapidly.
Get to know the principle of Concourse if you are not familiar with the Concourse CI tool. For more information, see Concourse official website.
Log on to the Container Service console to create a cluster. In this example, create a swarm cluster with one node.
For how to create a cluster, see Create a cluster.
A fully-managed service compatible with Kubernetes to help users focus on their applications rather than managing container infrastructure
A high-performance container management service, based on Docker and Kubernetes, providing containerized application lifecycle management
Alibaba Cloud CDN: The Best Choice When You Enter the APAC Market
2,605 posts | 747 followers
FollowAlibaba Clouder - February 9, 2021
Alibaba Cloud Native Community - December 29, 2023
Alibaba Cloud Indonesia - May 29, 2023
Alibaba Clouder - June 10, 2020
Alibaba Clouder - November 24, 2017
Alibaba Clouder - February 9, 2021
2,605 posts | 747 followers
Follow
Auto Scaling
Auto Scaling automatically adjusts computing resources based on your business cycle
Learn More
ECS(Elastic Compute Service)
Elastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn MoreMore Posts by Alibaba Clouder
