This topic describes the benefits of X-Engine. X-Engine provides 50% lower storage costs, but can still ensure the same performance as InnoDB.

Background information

X-Engine is a storage engine that is developed by Alibaba Cloud to reduce the disk usage and overall database cost of your ApsaraDB RDS for MySQL instance. X-Engine stores data in a tiered storage architecture and uses the Zstandard algorithm to compress data at a high compression ratio. You can understand the advantages of X-Engine over InnoDB and TokuDB by comparing their storage overheads and performance.

Note The major technological innovations of X-Engine have been released at three top academic conferences: ACM SIGMOD 2019 and VLDB2020 in the database field and the USENIX Conference on File and Storage Technologies (FAST) 2020 in the storage field.

Test environment

The RDS instance used for testing is created with the rds.mysql.s3.large type and a storage capacity of 2 TB. This instance type supports four CPU cores and 8 GB of memory.

50% lower storage costs than InnoDB

Disk usage comparison between X-Engine and InnoDB

The preceding figure shows a comparison of disk usage between X-Engine and InnoDB.

Both X-Engine and InnoDB use their default configurations and the default table schema provided by SysBench. Each table contains 10 million data records, and the total number of tables gradually increases from 32 to 736. As the data volume grows, the disk usage of X-Engine increases at a lower speed and drops by up to 42% compared with InnoDB. In addition, X-Engine requires less disk space to store large-sized individual data records. For example, after an image database is migrated to X-Engine, it occupies only 14% of the disk space required in InnoDB.

InnoDB does not compress data in most of its business scenarios. If data compression is enabled, the disk usage of InnoDB decreases by about 33%, but the query performance also sharply decreases. For example, the performance for updates based on primary keys decreases by about 90%. This interrupts your business. X-Engine can compress data to reduce storage costs while maintaining stable performance.

Sysbench test command:

#InnoDBprepare Database
sysbench/usr/share/sysbench/oltp_update_index.lua\
    --mysql-host=[RDS instance connection string]\
    --mysql-user=sbtest\
    --mysql-password=sbtest\
    --mysql-db=sbtest\
    --threads=32\
    --tables=[32-736]\
    --table_size=10000000\
    --mysql-storage_engine=INNODB\
    prepare

#X-Engineprepare Database
sysbench/usr/share/sysbench/oltp_update_index.lua\
    --mysql-host=[RDS instance connection string]\
    --mysql-user=sbtest\
    --mysql-password=sbtest\
    --mysql-db=sbtest\
    --threads=32\
    --tables=[32-736]\
    --table_size=10000000\
    --mysql-storage_engine=XENGINE\
    prepare

Lower storage overheads than TokuDB

TokuDB no longer offers low cost storage options. Also, Percona no longer offers support, maintenance, and updates for TokuDB. However, X-Engine offers lower storage overheads than TokuDB. We recommend that you change the storage engine of your ApsaraDB RDS for MySQL instance from TokuDB to X-Engine.

TokuDB uses a fractal tree structure. This structure consists of more leaf nodes than the B+-tree structure that is used by InnoDB. These leaf nodes are populated with data records and stored as data blocks. Therefore, TokuDB supports a higher data compression ratio than InnoDB. However, the fractal tree structure does not support tiered storage. The tiered storage architecture of X-Engine not only consists of blocks that are populated with data records but also offers optimized storage. This allows X-Engine to support lower storage overheads than TokuDB.

Comparison of disk usage between X-Engine and TokuDB

The preceding figure shows a comparison of disk usage between X-Engine and TokuDB.

A total of 32 tables are created on the RDS instance used for testing. Each table contains 100 million data records. These data records occupy 411 GB of disk space in TokuDB and 400 GB of disk space in X-Engine. This proves the improvements and advantages of X-Engine over TokuDB in terms of storage costs.

Sysbench test command:

#TokuDBprepare Database
sysbench/usr/share/sysbench/oltp_update_index.lua\
    --mysql-host=[RDS instance connection string]\
    --mysql-user=sbtest\
    --mysql-password=sbtest\
    --mysql-db=sbtest\
    --threads=32\
    --tables=[32-736]\
    --table_size=1000000000\
    --mysql-storage_engine=TokuDB\
    prepare

#X-Engineprepare Database
sysbench/usr/share/sysbench/oltp_update_index.lua\
    --mysql-host=[RDS instance connection string]\
    --mysql-user=sbtest\
    --mysql-password=sbtest\
    --mysql-db=sbtest\
    --threads=32\
    --tables=[32-736]\
    --table_size=1000000000\
    --mysql-storage_engine=XENGINE\
    prepare

Tiered storage and tiered access to increase QPS

X-Engine reduces the disk usage for cold data to lower the overall storage cost while maintaining a stable rate of queries per second (QPS) for hot data.

  • The tiered storage architecture of X-Engine allows you to store hot and cold data at different tiers and compress cold data by default.
  • X-Engine applies technologies such as prefix encoding to every data record. This allows you to reduce storage overheads.
  • In most of the actual business scenarios, data is skewed because the volume of hot data is lower than that of cold data. Despite the skewed data, the tiered access architecture of X-Engine allows you to increase the QPS.
Performance of point queries

The preceding figure shows the performance of X-Engine for processing point queries to skewed data.

This test uses the popular method of Zipf distribution to control the degree of data skew. If the degree of data skew (namely, the Zipf factor) is high, more point queries hit hot data in the cache instead of cold data on disks. This decreases the access latency and increases the QPS. In addition, the compression of cold data only has a small impact on the QPS.

The tiered storage and tiered access architectures of X-Engine eliminate the impact of cold data on most of the SQL statements that are executed to query hot data. These architectures also increase the QPS by 2.7 times than when all data is evenly accessed.

Sysbench test command:

sysbench/usr/share/sysbench/oltp_point_select.lua\
    --mysql-host=[RDS instance connection string]\
    --mysql-user=sbtest\
    --mysql-password=sbtest\
    --time=3600\
    --mysql-db=sbtest\
    --tables=32\
    --threads=512\
    --table_size=10000000\
    --rand-type=zipfian\
    --rand-zipfian-exp=[0-1]\
    --report-interval=1\
    run

Equally matched performance of X-Engine and InnoDB for cold data queries

X-Engine and InnoDB offer an equally matched QPS and transactions per second (TPS) for processing queries to a large volume of cold data, especially archived and historical data.

Performance for processing queries to cold data

The preceding figure shows a comparison of cold data query performance between InnoDB and X-Engine.

In most of the online transactional processing (OLTP) scenarios, X-Engine and InnoDB offer equally matched performance for processing frequent updates (oltp_update_index and oltp_write_only) and point queries (oltp_point_select).

However, when X-Engine queries data for a specific time range or checks the uniqueness of a single data record, its tiered storage architecture requires a scan or access to a number of storage tiers. As a result, X-Engine processes range queries (oltp_read_only) and inserts (OLTP_insert) at a slightly lower speed than InnoDB.

X-Engine processes data reads and writes (oltp_read_write) at the same speed as InnoDB.

Sysbench test command:

#oltp_read_only
sysbench/usr/share/sysbench/oltp_read_only.lua\
    --mysql-host=[RDS instance connection string]\
    --mysql-user=sbtest\
    --mysql-password=sbtest\
    --mysql-db=sbtest\
    --time=3600\
    --tables=32\
    --threads=512\
    --table_size=10000000\
    --rand-type=uniform\
    --report-interval=1\
    run

Summary

X-Engine is a storage engine that is tailored to the cost-effectiveness requirements of ApsaraDB RDS for MySQL. It offers performance that is comparable to InnoDB but at lower storage costs. X-Engine has been used in a number of core businesses of Alibaba Group. These include the DingTalk chat history database, Taobao image database, and Taobao transaction history database.

Get started with X-Engine

If you are new to ApsaraDB RDS for MySQL, select X-Engine as the default storage engine when you create an instance. For more information, see Create an ApsaraDB RDS for MySQL instance.