Alibaba Cloud's cloud definition storage is coming

Cloud Defined Storage (CDS) officially released

On the occasion of the Mid Autumn Festival and National Day in 2021, Alibaba Cloud storage team released a new cloud defined storage CDS.

This release of CDS version 1.0, based on Alibaba Cloud Apsaras operating system and distributed storage base Pangu, is the first time in the industry to simultaneously support the object storage service OSS, block storage service EBS, log service SLS and disaster recovery service HBR, especially through Pangu's new QoS framework to support the integration and deployment of object storage service OSS and block storage service EBS on the same set of physical servers. The storage service released this time is basically consistent with the large-scale storage service on the public cloud in terms of code version, bridging the large gap between the previous hybrid cloud version and the public cloud version, so that customers can enjoy the functions that are basically consistent with the public cloud.

Why does Alibaba Cloud storage team release CDS? What is the difference between CDS and traditional storage array, distributed storage and software defined storage? How does the Alibaba Cloud storage team see the future development trend of storage? Here, the core technical leaders of our CDS R&D team will reveal the mysteries surrounding Alibaba Cloud CDS through a series of technical articles.

Why develop cloud defined storage?

Business requirements

Since Alibaba Cloud Storage established Hybrid Cloud Storage HCS (Hybrid Cloud Storage) at the end of 2016, there has been a lot of demand from internal and external businesses for distributed storage products with flexible scale configurations and competitive functions. Considering the investment in product research and development, Alibaba Cloud was still inexperienced in the hybrid cloud market, and there were many participating manufacturers and fierce competition in the domestic market. Our strategy was to start with a combination of distributed storage, arrays, and disaster recovery all-in-one machines integrated with proprietary cloud, and first enter different segments of the traditional storage market to test water, so as to help Alibaba Cloud understand and explore this market, store experience and ammunition.

Since 2018, as the pace of IT infrastructure cloudization has accelerated, cloud based infrastructure or data center has become the preferred solution for government and enterprise customers, bringing huge market opportunities to new storage vendors, especially software definition storage vendors. At the same time, with the development of Alibaba Cloud in the hybrid cloud market, the requirements for Alibaba Cloud storage output in the hybrid cloud scenario are increasing, Therefore, the storage team will officially start the planning, design and development of related products in 2019.

Tough decisions

After focusing on the direction of distributed storage and SDS, through analysis and research, we found that the homogeneity of domestic SDS products is very serious. If we break away from the public cloud technology stack and build a similar product, it is difficult to build a lasting core competitiveness. At the same time, from the perspective of technology and engineering, the parallel development of public cloud and private cloud product architectures will cause huge waste of resources, Many similar features need to be redone on different products, and can not be guaranteed to be supported due to architectural constraints. Finally, we decided to start the self research of hybrid cloud distributed storage SDS. Technically, we insisted that the public cloud and hybrid cloud CDS share the same set of technologies and codes, maintain the elasticity, scalability and differentiated competitiveness of service hosting of public cloud storage services, and actively complement the advanced features of enterprise storage. Therefore, the architecture, functions and user experience of hybrid cloud CDS are consistent with that of public cloud, which is the origin of the "cloud definition" of cloud definition storage.

Next, we will introduce the characteristics of software defined storage and the differences between cloud defined storage and software defined storage.

Software Defined Storage

Origin of software definition storage

The statement of software defined storage (SDS: Software Defined Storage) was first seen in a series of market campaigns in 2012 when VMware and EMC promoted the software defined data center (SDDC:, As well as the pressure and competition brought by AWS/Azure cloud computing vendors, the path and direction of enterprise data center and storage have been redefined for cloud computing. The hottest historical process of SDS in that year is shown in the following figure:

Key characteristics of software definition storage

Software definition storage has many characteristics, among which there are three recognized characteristics in the industry: abstract, pool and automation. The driving force behind these features is enterprise cloud and enterprise ICT infrastructure cloud.

• Abstraction, software and hardware decoupling, management and control decoupling: cloud computing manufacturers and cloud based data centers use a large number of standardized/commercialized servers to replace proprietary hardware storage devices such as EMC/NetApp/IBM, and large-scale procurement and separate bidding for storage/servers promote the decoupling of storage software and hardware. The decoupling of data plane and control plane also helps customers manage different hardware (data plane) of different manufacturers through a unified control plane.

• Pooling and scale out: In order to improve resource utilization efficiency, storage virtualization technology is used to pool various storage and data stacks.

• Automation: Through the development of API, software can control the addition, deletion, modification and query of storage resources and services, which is more efficient than the human-computer interface and has stronger business flexibility.

Software Defined Storage vs. Traditional Storage Arrays/Software

SDS and traditional storage arrays are different in hardware coupling, node/controller expansion mode and scale, interaction and management interfaces:

It is worth mentioning that the "software definition" of SDS in foreign countries emphasizes that SDS is rich in API/SDK and can be driven by software programming, while traditional storage has more human-computer interaction interfaces, mainly driven by administrators.

Cloud defines what storage is

Cloud definition storage CDS is a new software definition storage, which can optimize the output of software and hardware on the customized CDS storage server, and also supports the software output mode from X86 to various information innovation platforms. The core keyword is "cloud definition", which can be summarized as follows:

• Cloud Native

CDS is not a redeveloped product. It is to gradually import and publish various storage services of Alibaba Cloud's public cloud to the unified CDS storage platform according to product planning and market demand. The code and features of storage services in the internal CDS, such as OSS, are completely consistent with those of the public cloud; Externally, various cloud storage services integrated by CDS, from the console (GUI), command line (CLI) to Open API/SDK, are consistent with the public cloud, reducing the difficulty of using and integrating with the original Alibaba Cloud customers and partners.

At the same time, CDS also naturally supports the layered architecture of public cloud: cloud region AZ cluster.

• Cloud Scale

Compared with most distributed storage or SDS manufacturers, the biggest advantage of Alibaba Cloud CDS is that these CDS codes have been polished and verified by Alibaba Cloud's public cloud millions of customers and Alibaba Group's various businesses (such as the Double 11), supporting the pressure of large-scale and complex businesses. Taking the OSS storage service of CDS as an example, a single bucket can support trillions of objects in the production environment, reaching EB level capacity.

• Cloud Service

CDS provides multiple cloud storage services on the same storage platform (generally the most common are block storage and object storage services), and supports multiple tenants. The front desk can provide customer self-service like the public cloud, and the back desk can provide independent operation and maintenance as well as hosting operation and maintenance (local on-site or remote). Various basic storage services can be controlled and maintained through the CDS cloud management suite, and through the Pangu unified data base. At the same time, advanced storage services such as log services, disaster recovery services, data management and analysis services can be opened.

At the same time, in order to speed up enterprise cloud deployment, Alibaba Cloud Storage synchronously releases enterprise storage functions on the public cloud and CDS, such as EBS based enterprise block storage ESSD and HBR based disaster recovery backup.

• To Cloud

The storage data flows between CDS and cloud outside CDS through a network controlled and secure way.

• Cloud deployment (on Cloud)

CDS is directly deployed in the cloud environment as a cloud storage service to provide external services. The cloud environment here can be public cloud, local cloud, edge cloud, proprietary cloud and other cloud architectures.

• Hybrid Multi Cloud

In addition to the needs of external customers, many business units within Alibaba Group also have storage needs when exporting products or services. Based on these needs, they or their customers have a demand to avoid being locked by a supplier. Mixed cloud has become a standard configuration. Here, there are two different interpretations of hybrid multi cloud: one is Alibaba Cloud's hybrid multi cloud, including Alibaba Cloud's public cloud, hybrid cloud, local cloud, and edge cloud computing; The other is Alibaba Cloud and other cloud computing service providers.

CDS, based on the unified Pangu 2.0 distributed storage hybrid cloud, has realized the deployment of hybrid cloud, cloud connectivity and other polymorphic clouds:

Redefined property publishing mode

Alibaba Cloud CDS not only redefines the new baseline of hybrid cloud SDS, but also redefines the engineering model of Alibaba Cloud storage product and feature development. Since CDS 1.0, cloud storage service products that have been released on the CDS platform will be required to be released at the same time as public cloud and hybrid cloud scenarios. At the same time, the hybrid cloud output, especially the scenario without public network connection, depends on the technical capabilities of customers or on-site service personnel; Alibaba Cloud CDS products focus on polishing intelligent operation and maintenance, RAS, and automatic diagnosis capabilities, which are also fed back into public cloud storage services to improve system operation and maintenance efficiency and enhance system stability.

The redefined engineering mode of CDS has greatly accelerated the energy efficiency of the release of the hybrid cloud storage version. In the next version of CDS in this fiscal year, important products and features such as cloud storage gateway, new enterprise fast storage ESSD and file storage NAS will be rapidly introduced.

Expectation

Alibaba Cloud CDS is a cloud defined storage platform. The current version 1.0 of CDS supports multiple Alibaba Cloud storage products and data services (including block storage EBS, object storage OSS, log service SLS, and disaster recovery service HBR). In the next technical series, we will gradually introduce the features of various products and key modules in CDS, such as platform and control services, Pangu, EBS, OSS and SLS.

The release of Alibaba Cloud CDS1.0 represents that Alibaba Cloud storage has begun to enter the hybrid cloud storage market as an authentic cloud storage to meet the growing demand for cloud computing. As the leader of domestic public cloud and the challenger of hybrid cloud market, we have firm confidence and determination to promote the development of China's storage technology together with experts and friends from colleges and universities.