PolarDB:History of PolarDB-X

Background

In 2009, Alibaba's demand for data storage was growing exponentially alongside its business. To scale without proportionally increasing infrastructure costs, Alibaba launched the de-IOE campaign: a strategic shift away from proprietary commercial software toward open source and cloud-native technology. Databases were at the center of that shift.

TDDL — distributed middleware at scale (2009)

Keywords: large-scale deployment at Alibaba, pioneer of database and table sharding

On November 11, 2009, Alibaba launched Taobao Distributed Data Layer (TDDL), a distributed database middleware built for high-concurrency transaction systems.

TDDL started as a JAR file installed on clients, but it already used a three-layer topology — Matrix, Group, and Atom — to handle different sharding requirements and resolve scaling and availability bottlenecks.

By 2015, TDDL had become the unified database access standard across Alibaba. Currently, approximately 300,000 TDDL instances run across Alibaba's business units, covering payment, funds, instant messaging, and media.

Tested every year by the Double 11 Shopping Festival, TDDL demonstrated high stability, performance, throughput, and MySQL syntax compatibility at massive scale. Its success gave Alibaba the confidence to commercialize distributed database technology.

DRDS — cloud commercialization (2016)

Keywords: first paying customers, high-performance SQL engine

In early 2016, Distributed Relational Database Service (DRDS) gained its first paying customer. From there, the team focused on improving unit processing efficiency to help users reduce costs.

In 2017, DRDS shipped a high-performance distributed SQL engine built on three components — PlanCache, FastSQL, and custom underlying drivers. The engine delivered measurable gains:

• 300% improvement in batch writing, shard key selection, and read/write splitting

• 200% improvement in cross-database integration, distributed joins, and distributed transactions

DRDS also supported join operations across tables of different dimensions, in-memory secondary sorting, and in-memory function operations — reducing overall usage costs compared to earlier approaches.

For enterprise deployments, DRDS provided DRDS Sequence, read/write splitting, database account management, and backend O&M instruction sets.

The successful commercial use of DRDS marked the transformation of distributed database technology from internal incubation to market-oriented operations as well as the transition from distributed database middleware to distributed database systems.

PolarDB-X 1.0 — architecture upgrade

Keywords: brand rename, public service sector adoption

With optimal stability, ultra-high performance, and a wide range of enterprise-level features, DRDS was used in multiple crucial projects for social and economic development in China from 2018 to 2019 and helped accelerate the digital transformation of Chinese IT infrastructure. Since then, DRDS has become a key service in Alibaba.

DRDS was upgraded and renamed PolarDB-X. In the new name, PolarDB refers to Alibaba Cloud's portfolio of cloud database services, and X stands for "extreme."

PolarDB-X 1.0 introduced the ability to use PolarDB for MySQL clusters as storage nodes, enhancing I/O capability and improving flexible transaction performance. For public service sector and enterprise users, PolarDB-X 1.0 added security features including consistent backup and restoration, SQL flashback, and SQL audit.

PolarDB-X 2.0 — finance-grade distributed architecture

Keywords: finance-grade high availability, transparent distribution, HTAP, open source, multi-cloud deployment

PolarDB-X 2.0 is the Alibaba distributed database release with the most updates. It was redesigned from the ground up around three principles: transactional consistency, operational transparency, and deployment flexibility.

Transactional consistency

• Uses the Timestamp Oracle (TSO) protocol and Multi-Version Concurrency Control (MVCC) to guarantee transactional consistency across distributed nodes

• Data nodes (DNs) run an Alibaba-built MySQL branch based on X-Paxos, with three replicas and strong consistency

• Achieves zero Recovery Point Objective (RPO) during disaster recovery

• Provides global binary logging and global consistent backup

Transparent distribution

• Handles sharding automatically based on the transparent distribution concept

• Uses consistent hashing for partitioning

• Supports linear scaling of distributed databases

Deployment flexibility

PolarDB-X 2.0 runs in 13 countries and regions worldwide and supports four deployment forms: public cloud, private cloud, software, and DBStack.

The source code of PolarDB-X 2.0 is available in open source MySQL communities, enabling broader contributions to high-performance distributed database technology.

Going forward, PolarDB-X 2.0 will focus on localization support, hybrid transactional and analytical processing (HTAP) workloads, faster capacity changes, and multi-active disaster recovery.