AnalyticDB for MySQLis a petabyte-scale real-time data warehouse independently developed by Alibaba. It has been validated in ultra-large-scale and core business scenarios.
Overview
Since its first release within Alibaba in 2012, AnalyticDB for MySQLhas released nearly a hundred versions. It supports a wide range of online analytics services within Alibaba, including e-commerce, advertising, logistics, entertainment, tourism, and risk management. AnalyticDB for MySQLhas been commercially available since 2014 and is used across more than ten industries, including traditional large and medium-sized enterprises, government agencies, and numerous Internet companies.
AnalyticDB for MySQLis a cloud-native data warehouse that is meticulously engineered based on the concept and trend of unified database and big data technology.
Technical architecture
AnalyticDB for MySQLuses a cloud-native architecture with decoupled compute and storage and tiered hot and cold data. It supports high-throughput real-time data writes and strong data consistency, and handles mixed workloads of high-concurrency queries and high-throughput batch processing.
AnalyticDB for MySQLcombines the advantages of distributed computing, elastic computing, and cloud computing, with significant improvements in scalability, ease of use, reliability, and security. It fully meets the real-time data warehousing requirements of different scenarios, supports larger-scale concurrent access, faster read and write capabilities, and more intelligent hybrid query workload management, and enables more refined resource utilization and lower costs. This allows you to focus on your business and the value of your data.
In terms of hardware deployment, AnalyticDB for MySQLsupports Intel and AMD chips and the x86 instruction set.
Enterprise EditionandBasic Edition
Enterprise Editionprovides service recovery within seconds, supports cross-zone deployment, and offers automatic fault detection, removal, and replica rebuild. With three-replica storage and full and incremental backups, it delivers finance-grade data reliability. It also provides supporting tools for data migration, data synchronization, data management, data integration, and data security, so that you can focus on your business.
Basic Editionuses a single-replica architecture and offers the same features as Enterprise Edition. However, because the underlying storage has only one replica, it does not provide high availability. It is suitable for business scenarios that do not require high data availability and seek low-cost hot data storage.
Enterprise EditionandBasic Editionhave the following architectures:
Access layer
The access layer consists of linearly scalable Multi-Master coordinator nodes. It is responsible for protocol access, SQL parsing and optimization, real-time write sharding, data scheduling, and query scheduling.
The access layer improves development efficiency through unified billing units, unified metadata and permissions, unified development languages, and unified transmission links.
Compute layer
The in-house compute engine provides both MPP and BSP modes. The MPP mode is a stream computing mode and is not suitable for low-cost, high-throughput offline processing scenarios. The BSP mode divides tasks through a directed acyclic graph (DAG) and schedules them in batches. It supports large-scale data computing with limited resources and allows computed data to be written to disk. The compute engine can automatically switch modes: when a query cannot be completed within a certain period in the MPP mode, the system automatically switches to the BSP mode.
The open-source Spark compute engine is also provided to meet more complex offline processing and machine learning scenarios. The Spark compute layer and the storage layer are interconnected, and you can use compute layer resources to process storage layer data. Creating and configuring Spark resource groups is easier this way. With hardware acceleration provided by GPUs and FPGAs, we can achieve better cost-effectiveness.
The in-house storage engine is a distributed, real-time, strongly consistent, and highly available engine built on shared storage.Data is written in parallel in real time through data sharding and Multi-Raft technology, and hot and cold data is separated through a tiered storage strategy to effectively reduce costs. Combined with hybrid row-column storage and intelligent indexing, the engine delivers excellent performance.
Storage layer
Only one copy of full data is required to serve both offline and online scenarios.
Online analytics scenarios require data to be placed on high-performance storage media to improve performance, while offline scenarios require data to be placed on low-cost storage media to reduce storage costs. To meet the requirements of different scenarios, a copy of full data is first stored on low-cost, high-throughput storage media. Low-cost offline processing scenarios directly read from and write to data on the low-cost storage media, which reduces data storage and I/O costs while ensuring high throughput. Real-time data is then stored on ESSD cloud disks to guarantee row-level real-time performance. At the same time, indexes are built on the full data, and data access is accelerated through caching to meet high-performance online analytics scenarios with sub-100-millisecond latency.
The built-in data warehouse storage is open to external systems and does not lock in your data. It provides a unified Storage API layer and the Arrow data access format. It supports accessing internal data with open-source compute engines such as Spark and Presto, and provides complete support for data lake ecosystems such as Iceberg, Hudi, Delta Lake, and Paimon.
Historical editions
Data Warehouse Edition
Data Warehouse Editionis mainly used to process data in high-performance online analytics scenarios.
Data Warehouse Editionhas the following architecture:
Access layer
The access layer consists of linearly scalable Multi-Master coordinator nodes. It is responsible for protocol access, SQL parsing and optimization, real-time write sharding, data scheduling, and query scheduling.
Compute engine
The compute engine integrates distributed MPP and DAG execution capabilities. Combined with an intelligent optimizer, it supports high-concurrency and complex SQL mixed workloads. With the cloud-native infrastructure, compute nodes support elastic scheduling and can scale within minutes or even seconds based on business requirements, achieving efficient resource utilization.
Storage engine
The storage engine is a distributed, real-time, strongly consistent, and highly available engine built on the Raft protocol. It achieves parallelism through data sharding and Multi-Raft, reduces costs through tiered storage of hot and cold data, and delivers excellent performance through hybrid row-column storage and intelligent indexing.
On top of this three-layer architecture, the service provides recovery within seconds, supports cross-zone deployment, and offers automatic fault detection, removal, and replica rebuild. With three-replica storage and full and incremental backups, it delivers finance-grade data reliability. It also provides supporting tools for data migration, data synchronization, data management, data integration, and data security, so that you can focus on your business.
Data Lakehouse Edition
Based on Data Warehouse Edition, the lakehouse-integrated edition that meets both low-cost offline processing and high-performance online analytics is called Data Lakehouse Edition. Data Lakehouse Editionis fully upgraded in the five aspects of the entire data pipeline: collection, storage, computing, management, and usage.
Data Lakehouse Editionhas the following architecture:
Data source
The data pipeline AnalyticDB Pipeline Service (APS) can ingest data from databases, logs, and big data platforms at a low cost with one click.
Access layer
The access layer improves development efficiency through unified billing units, unified metadata and permissions, unified development languages, and unified transmission links.
Storage layer+Compute layer
It supports the in-house XIHE compute engine and XUANWU storage engine, as well as the integrated open-source Spark compute engine and Hudi storage engine. The open-source capabilities provide you with richer data analytics scenarios. The in-house and open-source engines can access each other, providing a more unified experience.
Storage layer: only one copy of full data is required to serve both offline and online scenarios.
Online analytics scenarios require data to be placed on high-performance storage media to improve performance, while offline scenarios require data to be placed on low-cost storage media to reduce storage costs. To meet the requirements of different scenarios, a copy of full data is first stored on low-cost, high-throughput storage media. Low-cost offline processing scenarios directly read from and write to data on the low-cost storage media, which reduces data storage and I/O costs while ensuring high throughput. Real-time data is then stored on separate elastic I/O units (EIUs) to guarantee row-level real-time performance. At the same time, indexes are built on the full data, and data access is accelerated through caching to meet high-performance online analytics scenarios with sub-100-millisecond latency.
Compute layer: the XIHE compute engine intelligently selects the computing mode. The open-source Spark compute engine meets a variety of scenarios.
The XIHE compute engine and the Spark compute engine together form the cornerstone of the AnalyticDB for MySQLcompute layer. They complement each other to ensure the high performance required by online analytics while covering a wide range of offline processing and advanced analytics scenarios, providing a one-stop cloud data warehouse solution.
The XIHE compute engine provides both MPP and BSP modes. The MPP mode is a stream computing mode and is not suitable for low-cost, high-throughput offline processing scenarios. The BSP mode divides tasks through a DAG and schedules them in batches. It supports large-scale data computing with limited resources and allows computed data to be written to disk. The XIHE compute engine can automatically switch modes: when a query cannot be completed within a certain period in the MPP mode, the system automatically switches to the BSP mode.
The open-source Spark compute engine added in Data Lakehouse Edition can meet more complex offline processing and machine learning scenarios. In Data Lakehouse Edition, the Spark compute layer and the storage layer are interconnected, and you can use compute layer resources to process storage layer data. Creating and configuring Spark resource groups is easier.