By Eric Li (Zhuanghuai), Head of Alibaba Cloud-native Storage Business
Portability, extensibility, and dynamic features are inevitable for high performing cloud-native applications. However, cloud-native applications also impose requirements on density, speed, and hybrid performance of cloud-native storage. These requirements translate into requirements for efficiency, elasticity, autonomy, stability, low application coupling, GuestOS optimization, and security, in addition to the basic capabilities of cloud storage.
It's critical to address challenges related to performance, elasticity, high availability, encryption, isolation, observability, lifecycle, and other aspects, which arise from the containerization, cloud migration, and storage of new enterprise load or intelligent workload. Just upgrading storage products is not enough, improving the cloud-native control and data planes to promote the evolution of cloud-native storage are also equally important.
The following section describes possible cloud-native storage problems, scenarios, and feasible solutions. And further, describes what cloud-native storage and cloud storage are capable of now and what else to expect in the future.
Huge data is processed as a batch in a high-performance computing scenario, where thousands of pods start simultaneously through Kubernetes clusters, and autoscaler scales out hundreds of ECS instances to read data from and write data to shared file systems.
Latency time increases under heavy load, with an increased spike in high latency and unstable reads/writes.
Read/write requests of 10 Gbit/s in size are distributed to the same storage cluster in a high-performance computing scenario for centralized processing.
The bandwidth of the storage cluster is squeezed and affects the cluster access experience.
A large amount of biological data is processed and the number of files is small. However, the peak throughput is as high as 10 Gbit/s to 30 Gbit/s, with request density of 10,000 requests per second.
The occupied bandwidth almost reaches the bandwidth limit of the exclusive cluster.
Multi-host multi-card GPU training is performed. This is a read-intensive scenario as data in OSS is read directly.
Higher latency leads to a high IO wait value and GPU waits.
The tenant or application-level I/O metric monitoring and alerting are implemented in a multitenancy ZooKeeper or etcd environment.
Shared file systems or cache systems are created and deleted in a declarative manner.
In response to the preceding challenges in storage performance, elasticity, high availability, encryption, isolation, observability, lifecycle, and other aspects of a new computing mode, it's imperative to not just upgrade storage products, but also make improvements to the cloud-native control or data planes, to achieve stable, secure, autonomous, and efficient cloud-native storage v2 in the near future.
With effective improvements and enhancements on the cloud-native application layer, the storage cloud product layer, the underlying storage adaptation, and the storage core layer, it's possible to provide more stable, secure, autonomous, and efficient application-oriented cloud-native storage.
Let's take a look at a quick snapshot of the article:
The evolution of cloud-native storage v2 still requires a joint effort from the container team and the storage team, so that the storage capability in the cloud-native era is improved.
How to Locate Bottlenecks During Performance Tests and Address Occasional Timeouts?
376 posts | 41 followersFollow
ApsaraDB - December 21, 2022
Alibaba Developer - January 5, 2022
Aliware - November 10, 2020
Alibaba Cloud Native Community - July 19, 2022
Alibaba Clouder - April 3, 2020
Alibaba Cloud Community - December 21, 2021
376 posts | 41 followersFollow
Provides scalable, distributed, and high-performance block storage and object storage services in a software-defined manner.Learn More
An encrypted and secure cloud storage service which stores, processes and accesses massive amounts of data from anywhere in the worldLearn More
Simple, scalable, on-demand and reliable network attached storage for use with ECS instances, HPC and Container Service.Learn More
Plan and optimize your storage budget with flexible storage servicesLearn More
More Posts by Alibaba Cloud Native Community