This topic covers the history of PolarDB-X.
Background
In 2009, while closing its annual financial books, Alibaba found that its demand for data access and storage resources was growing exponentially and outpacing key business metrics. Alibaba recognized that without technical innovation to balance cost and growth, its future development would be at risk.
This led to the de-IOE initiative. The de-IOE initiative was not just about moving away from traditional IT vendors. As the internet and cloud computing accelerated, enterprises urgently needed a more open, flexible, and efficient IT infrastructure to support their growth and innovation. In short, de-IOE drove the shift from vertical to horizontal scaling and from commercial to open source software, laying the groundwork for cloud computing.
Replacing traditional commercial databases was at the core of the de-IOE initiative.
TDDL phase
Keywords: large-scale adoption at Alibaba; pioneer of database and table sharding
After moving away from traditional commercial databases, Alibaba needed a better alternative. For databases, the answer was clear: open source and distributed. Open source addressed costs, while distribution solved performance and capacity challenges.
On November 11 of the same year, TDDL (Taobao Distributed Data Layer) was released, pioneering the use of distributed database middleware with open source databases for high-concurrency transaction systems.
At the time, TDDL was a client-side JAR file, but it introduced an innovative three-layer (Matrix, Group, and Atom) sharding topology. This allowed applications to define custom sharding policies while solving enterprise challenges like elastic scaling and local high availability.
From 2011 to 2015, TDDL became the unified access standard for database systems at Alibaba, providing distributed database services to all its business units. Today, about 300,000 instances run within the group, serving over ten major business categories including payments, finance, instant messaging, and media.
The diverse business models drove TDDL's excellent MySQL syntax compatibility. The massive scale of operations honed its core stability, and the annual Double 11 shopping festivals helped it achieve industry-leading performance and high throughput.
Meanwhile, the commercialization of Alibaba's distributed database technology quietly began.
DRDS phase
Keywords: cloud commercialization; high-performance SQL engine
In early 2016, DRDS acquired its first paying customer on the public cloud. Since then, the DRDS team continuously improved processing capacity per unit of resources, helping customers reduce costs and improve efficiency.
In 2017, the DRDS R&D team released a new-generation high-performance distributed SQL engine. By using PlanCache, FastSQL, and custom underlying drivers, it delivered a 300% performance improvement for operations like batch writes, selects with sharding keys, and read/write splitting. It also provided a 200% performance boost for cross-database aggregation, distributed joins, and distributed transactions.
DRDS lowered usage costs by supporting features such as join operations across tables with different dimensions, in-memory secondary sorting, and function computation on in-memory results.
For distributed database use cases, DRDS also provided a suite of enterprise-grade features, including a global sequence service, read/write splitting, a database account system, and a set of DRDS backend O&M commands.
With its excellent performance and a solid user experience, DRDS quickly built a loyal customer base on the public cloud.
The commercial success of DRDS marked a milestone: Alibaba's distributed database technology moved from internal incubation to market-oriented operations and made a substantial leap from distributed database middleware to a complete distributed database system.
PolarDB-X 1.0 phase
Keywords: architecture and brand upgrade; national strategic projects
From 2018 to 2019, DRDS was adopted for many national strategic projects in government and enterprise sectors due to its excellent stability, ultra-high performance, and rich enterprise features. It played an active role in the digital transformation of China's IT infrastructure, enhancing its brand reputation and becoming a flagship product for Alibaba.
During this time, DRDS underwent a brand upgrade and was renamed PolarDB-X. "PolarDB" is the family name for Alibaba Cloud's self-developed relational database products, and "X" sounds like "extreme" and signifies "ultimate".
PolarDB-X 1.0 supported the use of PolarDB for MySQL as storage nodes, significantly increasing cluster I/O capabilities and enabling flexible distributed transactions. It also enhanced security features for government and enterprise customers, such as consistent backup and restoration, SQL flashback, and SQL audit.
PolarDB-X 2.0 phase
Keywords: financial-grade high availability, transparent distribution, integrated centralized and distributed architecture, HTAP integration, open source, and multi-cloud
PolarDB-X 2.0 is the most significant version update in the history of Alibaba's distributed databases. Based on the concept of transparent distribution, the product provides a default primary key sharding policy, high-performance and strongly consistent distributed transactions based on TSO and MVCC, linear scale-out based on a consistent hashing partitioning strategy, and globally consistent binlog and backup capabilities. Its storage nodes (DNs) use an X-Paxos-based, three-replica, strongly consistent MySQL branch developed by Alibaba to ensure an RPO of 0 during disaster recovery.
PolarDB-X 2.0 operates on a new architecture, providing services in 13 countries and regions worldwide through Alibaba Cloud. It is available in multiple deployment forms, including public cloud, private cloud, software, and DBStack, to meet the needs of customers across different industries.
PolarDB-X 2.0 has fully embraced open source, growing with the global MySQL community to improve high-performance distributed databases and lower the barrier to adoption.
In the future, PolarDB-X 2.0 will continue to explore areas such as domestic market adaptation, HTAP hybrid workloads, rapid capacity scaling, and multi-region disaster recovery and active-active solutions.