By OpenAnolis Kernel SIG
CPU Co-location refers to deploying different types of services to run on the same machine, allowing them to share the CPU resources on the machine to improve CPU utilization, thereby reducing the purchase and operation costs of the machine. However, some types of tasks are very sensitive to latency (such as e-commerce, search, and web services). Such tasks require higher real-time performance but usually consume fewer resources, which are called online tasks. There are also tasks that focus more on computing or batch processing and are not sensitive to latency but consume relatively more resources, which are called offline tasks.
When these two types of tasks are deployed on the same machine at the same time, due to the high resource consumption of offline tasks, the latency of online tasks is greatly affected by resource competition. In addition, on machines with hyper-threading architecture, even if offline tasks and online tasks run on different hyper-threading CPUs, the competition between pipelines and caches will affect the execution of online tasks. Therefore, CPU Co-location technology was created to solve the impact of offline tasks on online task latency and improve the utilization of CPU resources.
(Figure 1: Schematic Diagram of the Standalone CPU Utilization of Co-location)
The CPU Co-location technology, which is mainly implemented by the standalone operating system scheduler, determines the allocated CPU resources by task type. The three major native technologies of OpenAnolis provide the Koordinator community with powerful underlying technical support for CPU Co-location, including:
Group Identity (a CPU Co-location technology of OpenAnolis) provides the CPU Co-location capability to the operating system kernel. For example, Alibaba Cloud Linux 2/3 and Anolis 7/8 OS distributions use this technology. Group Identity adds another run queue to the original CFS to distinguish between online and offline tasks. In addition, Group Identity evicts offline tasks to avoid interference on the peer CPU (hyper-threading architecture). The Group Identity technology of OpenAnolis has been verified by large-scale activities (such as Alibaba’s Double 11 Shopping Festival and large-scale commercialization), and its CPU Co-location capability has been recognized by most users and developers.
Koordinator standalone operating systems are mainly Alinux 2/3 and CentOS 7.9. For the former, its CPU Co-location technology comes from the Group Identity of OpenAnolis, but for CentOS 7.9, its kernel currently does not provide the CPU Co-location capability. There may be several solutions for this situation:
For the first solution, you need to download the kernel source code from the CentOS image site, port the CPU Co-location technology to the kernel, compile it, install it, and restart the system to use this technology. However, this process will involve business migration and downtime, which will inevitably be costly to the business side.
For the second solution, although the migration workload is large, Alinux2/3 or Anolis OS provides a complete resource isolation solution of Co-location (CPU Co-location is only one of them), and the benefits of technical dividends are much greater than the migration costs. Moreover, CentOS is about to stop serving. OpenAnolis launched the Anolis OS distribution to solve this problem. This operating system is compatible with CentOS, and users can perform migration seamlessly.
In view of the lack of CPU Co-location capability of Koordinator cloud-native CentOS standalone operating system, OpenAnolis developers have given another solution that provides a scheduler plug-in package with CPU Co-location technology using the live upgrade technology of plugsched scheduler. The plug-in contains Alibaba Cloud's early (2017) CPU Co-location technology bvt + noise clean and can be installed directly to CentOS 7.9 without downtime and business migration, etc.
The live upgrade of the plugsched scheduler is a live upgrade development tool for the plugsched SDK scheduler provided by OpenAnolis. First, it can decouple the scheduler from the Linux kernel to form an independent module. Second, it can transplant the CPU Co-location technology to the scheduler module to form a scheduler plug-in. Finally, it can install the scheduler plug-in in the running system, and you can use the CPU Co-location technology. Plugsched can add, delete, and change kernel scheduler features dynamically to meet business needs without business migration and shutdown upgrades. It also supports rollback. Kernel developers can use the plugsched SDK to produce various types of scheduler plug-ins to meet different business scenarios.
The paper entitled Efficient Scheduler Live Update for Linux Kernel with Modularization has been included in the ASPLOS. It details the principle and application value of the plugsched technology and provides a comprehensive test and evaluation. Plug-ins produced by plugsched have been deployed in Ant Group, Alibaba Cloud, and a large domestic internet enterprise.
Open-Source Plugsched: https://gitee.com/anolis/plugsched
The Group Identity Co-location technology was tested in the offline best practice manual of the Koordinator community. The online task is the Nginx service, and the offline task is ffmpeg video transcoding. The machine adopted is Alibaba Cloud ECS Bare Metal Instance. The system release version is Alibaba Cloud Linux 2, and the system kernel is Alibaba Cloud Kernel 4.19, where the Group Identity CPU Co-location technology of OpenAnolis is installed. The test case is listed below:
Baseline: Run the Nginx container separately
Control Group: Run the Nginx container and ffmpeg container at the same time without setting the priority
Experimental Group: Set Nginx as an online high-priority task and ffmpeg as an offline low-priority task
Pressure Tester: Use the wrk tool on another server to send a request to the Nginx service
Test results (unit: ms)
| Baseline | Control Group | Experimental Group | |
| RT-P90 | 0.533 | 0.574 (+7.7%) | 0.548 (+2.8%) |
| RT-P99 | 0.93 | 1.58ms (+70%) | 0.96 (+3.2%) |
| CPU% | 29.6% | 85.6% | 64.8% |
(Data Source: Best Practice Manual for Offline Co-location of Alibaba Cloud ACK)
As shown in the control group, when the online and offline tasks run at the same time, the long tail latency of the online task Nginx is significantly affected, especially the P99 latency, which increases by about 70%. However, after priorities are set for online containers and offline containers (the Group Identity Co-location function is enabled), the long tail latency of P90 and P99 of Nginx is slightly disrupted, and the latency only increases by 2.8% and 3.2%.
The developers perform the same test on the scheduler plug-in to compare the performance difference between the CPU Co-location plug-in and the Group Identity. The Server configuration includes the following:
Test Machine: Alibaba Cloud ECS Bare Metal Instance
System Configuration: CentOS 7.9, Kernel version 3.10, and CPU Co-location scheduler plug-in installed
The configurations and pressures of the online and offline containers are the same as those in the test on Group Identity. The test results are listed below: (unit: ms)
| Baseline | Control Group | Experimental Group | |
| RT-P90 | 0.444 | 0.575 (+29.50) | 0.504 (+13.51%) |
| RT-P99 | 0.706 | 1.7 (+140.79) | 0.88 (+24.64%) |
| CPU% | 25.15% | 71.7% | 49.15% |
From the results above, without the CPU Co-location plug-in, offline tasks have a great impact on online tasks, and the P99 latency increases by more than double. However, after the CPU Co-location plug-in is installed, the impact on the P99 latency is significantly reduced, and the CPU utilization is close to 50%. However, its performance is not as good as Group Identity. Group Identity can enable online tasks to be interfered with by offline tasks by less than 5% (the interference is more than 20% if the CPU Co-location plug-in is used), and the CPU utilization can reach more than 60% (the CPU utilization is nearly 50% if the CPU Co-location plug-in is used).
The preceding test data shows that the performance of Group Identity is significantly better than the bvt + noise clean plug-in, regardless of the interference caused by offline tasks on online tasks or the increase in CPU utilization. This is due to the different implementation principles of the two technologies: the noise clean technology uses the throttle mechanism. When the scheduler selects the next task, it will detect the task type on the peer CPU and the type of task currently being executed by the CPU. If online and offline tasks run at the same time, the offline task will be throttled, and the next task will be selected for scheduling. This technology ensures that online tasks are executed first and are not interfered with by offline tasks on the peer CPU. However, Group Identity adds a low-priority run queue to CFS. Each time the next task is selected for execution, the task from the high-priority queue is preferentially selected, and an eviction queue is added to evict offline tasks being executed on the peer CPU. From the perspective of solution design, Group Identity is more suitable for CPU Co-location scenarios.
From the perspective of technological evolution, the Group Identity technology of the OpenAnolis is still being maintained and updated, while the bvt + noise clean technology is no longer updated and iterated. Furthermore, OpenAnolis has a complete resource isolation scheme for co-location, and the next generation of CPU Co-location technology, Group Identity v2.0, is being developed, which will bring more powerful features and performance.
The live upgrade technology of the plugsched scheduler supports the cloud kernel 4.19 and 5.10 of OpenAnolis more comprehensively and will be continuously maintained. However, the bvt + noise clean CPU Co-location plug-in in this article is supported on the CentOS kernel 3.10, and this kernel does not belong to OpenAnolis. Therefore, plugsched does not support this kernel that much, and its capability of live upgrade is not as good as those of the Anolis OS kernel. Support of Plugsched for kernel 3.10 is no longer updated and maintained.
Finally, technical personnel, open-source enthusiasts, readers, and users are welcome to experience the CPU Co-location technology, Group Identity, and Plugsched brought by OpenAnolis. We believe that whether it is Anolis OS, Cloud Kernel, or Plugsched, it will bring unexpected benefits and value to all of you. We welcome extensive technical personnel, open-source enthusiasts, readers, and users to join OpenAnolis and progress and develop with us.
Cloud Kernel SIG Homepage: https://openanolis.cn/sig/Cloud-Kernel
OpenAnolis White Paper: Using the EROFS Read-Only File System across the Cloud-Edge-End
83 posts | 5 followers
FollowAlibaba Cloud Native Community - December 1, 2022
Alibaba Cloud Native Community - June 29, 2023
Alibaba Cloud Native Community - December 7, 2023
Alibaba Cloud Native Community - March 29, 2023
OpenAnolis - January 24, 2024
Alibaba Cloud Community - December 2, 2022
83 posts | 5 followers
Follow
Function Compute
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn More
Elastic High Performance Computing Solution
High Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.
Learn More
Quick Starts
Deploy custom Alibaba Cloud solutions for business-critical scenarios with Quick Start templates.
Learn More
Cloud-Native Applications Management Solution
Accelerate and secure the development, deployment, and management of containerized applications cost-effectively.
Learn MoreMore Posts by OpenAnolis
