×
Community Blog Cluster Images: Achieve Efficient Distributed Application Delivery

Cluster Images: Achieve Efficient Distributed Application Delivery

This article explains cluster images and Sealer.

By Zhongyi

The Definition of Cluster Image

As the name suggests, similar to an operating system .iso image or Docker image, the cluster image is a resource package that combines all files in a cluster with a certain format using specific technical means.

1

Some interesting features can be found when comparing a single machine and a cluster:

  • The single machine contains drivers, such as computing, storage, and network, while a cluster contains drivers, such as CNI, CSI, and CRI.
  • The single machine runs the Ubuntu CentOS operating system, while the Kubernetes cluster can be considered as a cloud operating system in a cluster.
  • The single machine can run Docker containers or virtual machines, which is equivalent to a running instance. There are also instances running Kubernetes in the cluster.
  • Virtual machine images and Docker images are available on a single machine. With the development of cloud computing, similar image technologies are also abstracted on clusters.

Let’s take a Kubernetes-based cluster image as an example. The image contains all files except the operating system, including:

  • The binary variables on which Docker depends, system configuration, dockerd configuration, and a private container repository.
  • Binary Kubernetes core components, container images, and kubelet system configurations
  • The YAML configuration or helm chart for the application, and the container image of the application
  • All dependencies are required for running the application, such as other scripts, configurations, and binary tools.

Similarly, when a cluster image is running, it is not installed on a container or a machine. Instead, the image can be installed on multiple servers or the infrastructure of the public cloud directly.

An Introduction to Sealer

Sealer is an implementation method of Alibaba's open-source cluster image. Click here for the project address.

Docker solves the problem of mirroring a single container, while sealer builds, shares, and runs distributed software by packaging the entire cluster!!!

Imagine that we will deliver a SaaS application that relies on databases and middleware, such as MySQL, ES, or Redis, and everything is orchestrated on Kubernetes. If there is no cluster image, the following operations will be done:

  1. Use a tool to install the Kubernetes clusters
  2. Helm install MySQL/ES/Redis. For an offline environment. We may also need to import container images.
  3. Kubectl applies your SaaS.

This process may not seem complicated. However, from the perspective of project delivery, the preceding operations are process-oriented and error-prone.

Now, if another method was provided to solve the problem above with only one command, would you like to use it?

sealer run your-saas-application-with-mysql-redis-es:latest

We can see that only one cluster image needs to be run, and the entire cluster is delivered. By doing so, all detailed and complex operations are blocked, and any applications can run in the same way. How did the cluster image come from?

2

As shown in the figure above, we only need to define a file like Dockerfile, which is called Kubefile, and then run the build command below:

sealer build -t your-saas-application-with-mysql-redis-es:latest

It is clear when comparing a single machine with a cluster.

3

  • Docker builds a Docker image through Dockerfile and uses compose to run the container.
  • Sealer builds a CloudImage with Kubefile and uses Clusterfile to run the entire cluster.

Quick Experiment

Let's create and run a cluster image from the Kubernetes dashboard together to experience the complete process.

Write the Kubefile:

# Base image, which has been created with all Kubernetes startup-related dependencies
FROM registry.cn-qingdao.aliyuncs.com/sealer-io/cloudrootfs:v1.16.9-alpha.7
# Download the official dashboard YAML orchestration file and use the COPY command once it has been downloaded
RUN wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.2.0/aio/deploy/recommended.yaml
# Specify the running methods, such as kubectl, helm, kustomiz
CMD kubectl apply -f recommended.yaml

Build a dashboard cluster image:

sealer build -t kubernetes-with-dashobard:latest

Run the cluster image:

# The following command will install the Kubernetes cluster on the server and apply the dashboard. You can use passwd to specify the server ssh password, or alternatively use a key
sealer run kubernetes-with-dashobard:latest \
  --master 192.168.0.2,192.168.0.3,192.168.0.4 \
  --node 192.168.0.5,192.168.0.6 \
  --passwd xxx
# Check the pod
kubectl get pod -A |grep dashboard

Push the prepared image to the repository. The image is also compatible with the Docker registry:

sealer tag kubernetes-with-dashobard:latest docker.io/fanux/dashobard:latest
sealer push docker.io/fanux/dashobard:latest

By doing so, we can deliver the prepared image or make it available for reuse.

Scenarios

What specific things can Sealer help us do? The following section explains some main scenarios:

1. Install Kubernetes and Cluster Lifecycle Management, Including Upgrade, Backup, Restore, and Scale

This is the simplest scenario. You can use Sealer to install whether you need to install a development test environment on a single machine or install a high availability cluster in a production environment, whether it is bare metal, docked to the public cloud, or a variety of architecture operating systems. If you only want to install Kubernetes, you can just choose a base image.

Compared with other installation tools, Sealer has the following advantages:

(1) It is very simple that Sealer only runs one command.

(2) The speed is so fast that Sealer installs six nodes in three minutes. Sealer may be installed before you have finished downloading the nodes with other tools. Furthermore, it can be optimized to within two minutes (or even one minute) with advanced technology.

(3) Sealer has compatibility and stability that is compatible with various operating systems and supports architectures, such as x86 and ARM.

(4) Sealer has a consistent design, ensuring that the cluster stays defined in the Clusterfile. For example, the upgrade can be done by simply changing the version number in the Clusterfile.

Sealer’s fast speed is because of the Golang implementation. It means that we can do concurrent processing in various detailed places, which has more advantages than Ansible. Moreover, it supports more detailed error handling. Then, the previous load method is discarded on image distribution, and file distribution will be optimized to maximize installation performance.

For compatibility, Docker Kubelet uses the binary system plus systemd to install the core components into full containerization. This eliminates the need to rely on installation tools for perceptive operating systems, such as yum or apt. ARM and x86 adopt different image supports decoupled from Sealer. Adaptation to the public cloud is also implemented by extracting a separate module. Here, we did not connect to Terraform for performance. In our scenario, the startup of the infrastructure in Terraform was about three minutes, but we optimized it using backoff retry to less than 30 seconds. In addition, in the cluster image scenario, there is no need for such complex infrastructure management capabilities since we do not want to make Sealer heavy or rely on a command-line tool.

The consistent design is worth mentioning in Sealer. Cluster images and Clusterfile determine how a cluster looks. The same image and Clusterfile can run the same cluster. The Clusterfile can be changed by adding nodes, changing node specifications, or changing images. The layer with the same hash value will not be changed when changing images since the cluster image is also a hierarchical structure, and a change in the hash will help re-apply the layer.

2. Packaging and Installation of Cloud-Native Software, Such as Prometheus MySQL Clusters

Sealer can create a cluster with Prometheus or install Prometheus in an existing cluster by running prometheus:latest.

What is the difference between the helm and sealer?

(1) Sealer pays more attention to packaging rather than orchestration. In the preceding example, Prometheus can use helm for orchestration. Then, Sealer packages the chart and all the container images needed in the chart. This is achieved through advanced technologies during the build process. The build process starts a temporary Kubernetes cluster like Docker build. Then, which container images the cluster depends on will be known. The final step is to package container images.

(2) Sealer may not be able to install successfully with a chart by packaging with Kubernetes. For example, Sealer uses the abandoned API version, but the image is also packaged with Kubernetes. As long as the build is fine, the run will be fine. This is similar to the way Docker packages the operating system rootfs together.

(3) Sealer is integrated, so cluster images focus on the overall packaging of the entire distributed application cluster. For example, the Prometheus ELK MySQL cluster is made into an image service and business.

Therefore, Sealer and helm are collaborative, with a clear division of work.

These common cluster images can be found in the official repository of Sealer later and can be applied to users directly.

3. Overall Packaging and Delivery of SaaS Software, Offline Delivery of Alibaba Cloud Apsara Stack

From the perspective of distributed applications, in some situations, there can be components in the single digits, but in other situations, there can also be components in the hundreds. The entire delivery methods are process-oriented and require intervention, while Sealer can pack all these things together for one-click delivery.

Can't we add Ansible scripts to tar.gz to enable one-click delivery? The answer is yes. Just as before the Docker image appeared that everyone delivered through tar.gz, you will find that the emergence of standards and technologies has solved the problem of collaboration between users. With Cluster images, users can reuse the achievements of others and create things for others to use.

Sealer is suitable for Alibaba Cloud Apsara Stack scenarios. Many data centers of customers are offline, and cluster images put all dependencies in the image. Therefore, all local sites can carry out one-click delivery in the same way and obtain an excellent, consistent experience as long as the image is well done.

4. Implement Preceding Scenarios on the Public Cloud

Sealer can integrate with the public cloud. In many cases, a better user experience will be given if it is connected to the public cloud. For example, when installing a cluster, users only need to specify the number and specifications of servers without caring about the IP address and then modify the numbers defined in the Clusterfile directly.

An Introduction to Technical Principles

1. Copy-on-write

The storage of cluster images is also achieved by copy-on-write. This provides two advantages. On the one hand, users can deploy different distributed software in the same cluster at different layers to achieve reuse. On the other hand, users can push cluster images to the Docker repository directly.

2. Container Image Cache

During the build process, how does Sealer know which container images are in the cluster image to be built, and how can it store the container images? There are some difficult problems during the process:

(1) How do you know which container images are available in distributed software? These images need to be cached. Scanning the user's YAML file or scanning after with the helm template is always imperfect. First, there is no way to determine what user's orchestration. Since some software does not write the image address in the orchestration files but pulls it up through their own programs, there is no guarantee that a successful build run will be sufficient.

(2) A container image needs to be stored in a private repository and packaged in a cluster image. The address of the container repository is different from the container image in the orchestration file. In particular, how do you ensure users can still download the image in the private repository when users pull image.

There is a solution to the first problem. Like in a Docker build, a temporary Kubernetes cluster will be pulled up during the process of Sealer build, and the apply command defined in the Kubefile will also be executed.

4

As shown in the preceding figure, this ensures that all images that users depend on are packaged, regardless of the orchestration methods used.

The second problem is learning how to use this private image when packing the container image into a private repository. Let’s suppose that the private repository is named localhost:5000. This will be inconsistent with what is written in the orchestration file. Therefore, two solutions exist to solve this problem:

  • One solution is the hack and docker. It gives the ability to pull directly from private images as long as they are available in the private repository. It can only pull images from the public network if they are not available.
  • Another solution is proxy without Docker. Give all Docker requests to the proxy and let the proxy decide to pull from the private repository if it is available. At the same time, the registry has also been enhanced to allow it to cache multiple remote repositories.

5

Such Sealer solutions solve the problem of image packaging offline perfectly.

3. Load Balancing

The high availability of Sealer's cluster uses a lightweight load balancer lvscare. Compared with other load balancers, lvscare is very small, with only a few hundred lines of code. In addition, lvscare only guards the IPVS rules without loading. They monitor the apiserver on nodes. If it fails, lvscare will remove the corresponding rules and add them back automatically after starting again. In other words, it is equivalent to a dedicated load balancer It has been used in sealos for more than two years and has extensive practice.

6

4. Runtime

The runtime environment supports the running of applications. For example, the Sealer runtime based on Kubernetes can be supported in a very simple and transparent way. Let’s take istio as an example, users only need to do the following:

FROM kubernetes:v1.18.3
RUN curl -L https://istio.io/downloadIstio | sh -

Users can build an istio runtime for their applications by following the nodes above.

If it is not a runtime based on Kubernetes, such as k0s and k3s, users can extend the interface in the Sealer.Runtime:

FROM k3s:v1.18.3
RUN curl -L https://istio.io/downloadIstio | sh -

For better extensions, such as extending the ACK runtime:

FROM aliyum.com/ACK:v1.16.9
RUN curl -L https://istio.io/downloadIstio | sh -

This image helps users deploy applications in ACK. Some of the above are used in the roadmap.

5. Infrastructure

Now, many users want to run their cluster images on the cloud. Sealer comes with the ability to dock to the public cloud, and it can also implement its own infrastructure manager, which benefits from a more refined back-off retry mechanism. The performance of 30 seconds to complete the infrastructure build (six nodes in Alibaba Cloud) is the best in its class, and the number of API calls is reduced significantly with a Clusterfile-compatible configuration.

Summary

There are several future visions and values of Sealer:

  • Sealer allows users to customize clusters in an extremely simple way, solving the problem of collaboration between distributed software producers and users.
  • Sealer has an extremely simple and user-friendly interface shield and is compatible with various underlying technical details, running everywhere.
  • Sealer can be eco-built, and the official repository will cover commonly used distributed software.

Finally, it can be summarized:

  • If you want to deliver your distributed SaaS as a whole, please use Sealer.
  • If you want to integrate multiple distributed services, such as message queues for databases or microservices, please use Sealer.
  • If you want to install a distributed application, such as a MySQL primary-standby cluster, please use Sealer.
  • If you need to install or manage a Kubernetes high-availability cluster, please use Sealer.
  • If you want to initialize multiple data centers to ensure the status of these data centers is consistent, please use Sealer.
  • If you need to implement the preceding scenarios on the public cloud, please use Sealer.

More information about sealer: https://github.com/alibaba/sealer

0 0 0
Share on

You may also like

Comments

Related Products

  • Super Computing Cluster

    Super Computing Service provides ultimate computing performance and parallel computing cluster services for high-performance computing through high-speed RDMA network and heterogeneous accelerators such as GPU.

    Learn More
  • ApsaraDB for MyBase

    ApsaraDB Dedicated Cluster provided by Alibaba Cloud is a dedicated service for managing databases on the cloud.

    Learn More
  • ACK One

    Provides a control plane to allow users to manage Kubernetes clusters that run based on different infrastructure resources

    Learn More
  • Link IoT Edge

    Link IoT Edge allows for the management of millions of edge nodes by extending the capabilities of the cloud, thus providing users with services at the nearest location.

    Learn More