By Baotiao
In the evolution of cloud-native databases, decoupled storage and compute has allowed the compute and storage layers to scale independently, and shared storage has enabled multiple compute nodes to access a single copy of the data.
In "CloudJump: Optimizing Cloud Databases for Cloud Storages" (VLDB 2022), we analyzed how a database must redesign its I/O path once the storage medium changes from local disk to cloud storage;
In "CloudJump II: Optimizing Cloud Databases for Shared Storage" (SIGMOD 2025), we introduced Multi-Version Data (MVD) to address data consistency across multiple compute nodes in a shared-storage architecture.
This line of work leads naturally to the next question:
Cloud storage has never been a single medium. It is inherently a hierarchy — from local NVMe SSDs, to Enterprise SSD (ESSD) network block storage, to Object Storage Service (OSS) with virtually unlimited capacity — and each tier sits at a different point on the cost, latency, and throughput curves.
How can a database truly exploit that hierarchy — keeping hot data on fast media and cold data on cheap media — while holding performance steady and preserving correctness? That is the question CloudJump III answers.
The paper has been published at SIGMOD 2026 and is available for download: "CloudJump III: Optimizing Cloud Databases for Tiered Storage"
Modern cloud databases run on top of a tiered storage stack. For typical cloud-vendor offerings, the latency and cost of each tier are roughly as follows:
The closer a tier sits to compute, the lower its latency and the higher its bandwidth — and the more it costs. As user data volume keeps growing, storage cost (DRAM included) has become the dominant component of a database service's total cost of ownership (TCO). So the question is no longer "should we tier?" but "how should we tier?": under dynamically changing OLTP workloads, how should a data placement policy keep both performance and cost stable and predictable?
Separating hot from cold data is not a new idea, and the industry has built plenty of machinery around it: host-side NVMe caches (dm-cache, bcache, LVM Cache), information lifecycle management (ILM) and hierarchical storage management (HSM) pipelines, automatic tiering in storage arrays and cloud services, and designs built on object storage that keep only the hottest blocks on SSD. All of them share one trait: they operate outside the database kernel — at the block layer, the file system layer, or a gateway. They can observe I/O patterns, but they cannot see the internal state of the storage engine, and that state is exactly what correct, efficient tiering depends on. Tiering outside the kernel therefore misjudges the workload systematically:
The hottest pages sit in the DRAM buffer pool and generate almost no external I/O, so I/O-trace-based analysis makes them look "cold";
Structurally important index pages and metadata pages get demoted to slower media too early;
ILM/HSM works at too coarse a granularity: one small read triggers a recall of an entire file or a large extent, and periodic bulk migration contends with foreground I/O for bandwidth, producing recall storms;
Tiering data movement is decoupled from the kernel's recovery and backup protocols; because it does not cooperate with LSN semantics or the snapshot protocol, correctness is hard to guarantee in zero-downtime scenarios.
Once data is tiered, every fast-tier miss is punishing: the target page is either temporarily unavailable or must be fetched synchronously from a tier orders of magnitude slower, injecting uncontrolled jitter into queries. Manual time partitioning and table-level reorganization are both coarse-grained and reactive, and cannot keep up with workloads that shift minute by minute.
What operations teams actually need, then, is a tiering mechanism that keeps the performance and cost benefits of multi-tier storage, bounds the cost of a miss, and adds no operational fragility.
The core idea of CloudJump III is to move tiering decisions into the storage engine: data placement decisions are made at the buffer manager's natural control points (eviction and flush), using metadata only the engine can see — page type and lifetime, page-table relationships and quotas, temporary-table identification, dirty-page state, and buffer pool residency — to drive admission and write routing. We call this design eviction-centric and engine-aware.
The overall architecture splits into four layers, with an explicit boundary between the cache layers (volatile) and the storage tier (durable):
InnoDB Buffer Pool (DRAM):
The conventional in-memory cache holds hot pages and triggers tiering decisions at the natural control points of eviction and flush;
Buffer Pool Extension (BPE, local SSD):
A read-only extension of the buffer pool that caches clean pages only; it belongs to the volatile cache layer, so losing it does not affect durability;
OSS Buffer (ESSD):
The front end of the storage tier. It absorbs flushes of dirty pages and aggregates page-granularity updates into block-level writes, persisting data locally before it is synchronized to the remote tier. This both reduces write amplification and shields object storage from small random I/O;
OSS object storage:
The final durable tier: low cost, unlimited capacity, accessed via HTTP PUT/GET.
The read path is a tiered pipeline: DRAM → BPE → OSS Buffer → remote OSS, falling back tier by tier and promoting a page upward on a hit.
The write path is a multi-stage pipeline. Dirty pages flushed from the buffer pool first land on the ESSD behind OSS Buffer (near-side persistence); background threads then merge dirty pages into 2 MB objects and asynchronously PUT them to remote OSS in an append-only, versioned form. Each OSS-backed table is logically divided into fixed-size 2 MB objects (128 pages of 16 KB), and OSS Buffer allocates one block per object. Its internal structure follows the page management logic of the InnoDB buffer pool: a page-to-block hash for fast lookup, LRU for replacement, and a flush list to track pending write-back I/O.
When local SSD serves as an extension cache for the buffer pool, its worst enemy is cache pollution: large numbers of pages touched only once (one-hit wonders) get written to SSD indiscriminately, burning write endurance and crowding out the pages that would actually be reused. The BPE design borrows the selective admission and second-chance ideas of S3-FIFO:
In the DRAM buffer pool, we add a ghost list alongside the traditional young/old sublists, holding only the IDs of recently evicted pages (pure metadata, no data). A page evicted from the old sublist enters the ghost list first; only if it is accessed again while in the ghost list — proving it is worth reusing — is it promoted into BPE's SSD space. This deferred, reuse-verified admission naturally filters out one-hit pages. Inside BPE we use a modified CLOCK replacement algorithm, replacing the LRU list with a clock hand and a second-chance bit; this lowers synchronization overhead while aligning eviction order with the SSD layout, yielding more sequential writes and less wear.
The ablation study shows the effect directly: compared with full-admit, the ghost admission policy raises TPS by 12% and lifts the hit rate from 63.4% to 72.7%, while BPE IOPS drops from 22,369 to 2,082, about 91% lower — fewer fine-grained writes and smoother write-back.
Two engineering details of BPE stand out. First, metadata compaction: a 64 GB BPE instance needs only about 184 MB of control metadata and eliminates per-page lock structures. Second, chunk-based resizing: BPE space is divided into 64 MB chunks that can be added or removed online in seconds, so a hot instance can expand its cache temporarily and release it afterwards without a restart.
OSS Buffer's job is to turn the database's page-level writes into object-storage-friendly block-level writes. Flushing a dirty block runs through a three-stage asynchronous pipeline:
The block-level metadata is updated first (recording the version and marking the block in-flight); the block's dirty pages are then merged into a contiguous 2 MB segment and PUT asynchronously to OSS (each flush carries a monotonically increasing version number); on success, the block moves from Dirty to Clean and then to Free.
The key here is guaranteeing crash consistency. The same logical page may live in memory, in BPE, and in OSS Buffer at the same time, so CloudJump III ensures recoverability with a set of complementary write-ordering constraints: on a local write, metadata is updated before the data (recording intent); on remote synchronization, the data is transferred first and the metadata is marked clean afterwards. This crossed ordering guarantees that any block is either fully durable or safely replayable: if metadata was updated but the data is missing, recovery replays the redo; if the data is there but the metadata is stale, the block is simply flushed again. Both paths are idempotent.
Flush scheduling combines batch aggregation, a time delay (so hot blocks are not written over and over), watermark monitoring, and rate limiting (throttling throughput under OSS QPS limits and multi-tenant fairness constraints). A clean block becomes an eviction candidate only after its OSS synchronization completes, so space is always reused after persistence.
Once storage is tiered, backup and recovery correctness is both the easiest thing to overlook and the easiest thing to get wrong.
CloudJump III uses a snapshot versioning protocol to keep the evolution of tiering state in step with backup. The system maintains a global next_snapshot_version. When a backup starts, it briefly takes a lock and increments the version, and every subsequent OSS write carries that version. For each 2 MB object, if it is modified after the new snapshot epoch, a new version of the object is written out (for example, file_0002_v2) and the old version is kept for the backup to read. Each table maintains a .meta file recording each block's dirty bit and version number, and an incremental backup copies only the objects that changed since the last backup.
The backup flow follows a lock–version–copy sequence: lock the metadata to pin a version view, increment the snapshot counter, then make an in-bucket shadow copy through OSS's fast-copy API. Because fast-copy duplicates only object metadata and moves no data, snapshot creation finishes in seconds even for tables of several TB.
Measured numbers: CloudJump III creates a snapshot in 0.64 seconds, close to the 0.52 seconds of the ESSD-only path and far faster than the 53.73 seconds of the OSS-only path, while total backup and restore time matches the OSS baseline (the bottleneck is remote bandwidth, and tiering itself adds no extra overhead).
We evaluated CloudJump III on two configurations: a small one (8 cores, 1 GB buffer pool, 50 GB of data) and a large one (64 cores, 100 GB buffer pool, 5 TB of data), with fast-tier capacity covering 5% to 50% of the dataset. Workloads included Sysbench, YCSB (Zipf skew 0.8–2.0), Voter, and a real workload from an Alibaba Cloud production game customer (large BLOB updates plus bursty writes).
Several representative findings:
TPS rises monotonically with cache ratio and skew, with a clear knee once the hot data fits in the fast tier. Under moderate skew (Zipf α ≥ 1.4), a cache ratio of 20%–30% comes close to the ESSD-only throughput ceiling, while cost performance (CP) is markedly higher;
The Voter workload sustains 79%–80% of ESSD-only throughput at 64 threads, and the game workload stays within ±5% of the baseline throughout, even beating it by 3%–4% at moderate concurrency — large writes bypass straight to ESSD and avoid contending for the shared flush queue, while small updates are coalesced by OSS Buffer into 2 MB objects that smooth out bandwidth demand;
The ablation study confirms what each mechanism contributes: ghost admission (+12% TPS, −91% BPE IOPS), temporary-table exclusion (short analytical bursts no longer hit remote storage bandwidth), and DDL/BLOB bypass (which eliminates the deep drop in foreground throughput during DDL).
Three years of production deployment taught us a few things that go beyond the paper itself:
The fast tier should be a shared cache, not dedicated storage. Partitioning fast-tier capacity rigidly per table wastes capacity and destabilizes performance. CloudJump III reinterprets the fast tier as a globally shared cache governed by a unified eviction and quota policy. Logical ownership is decoupled from physical placement, and capacity is allocated dynamically as the workload shifts.
An eviction-centric design beats proactive migration. Architectures that move data with prediction models and background controllers cost far more in engineering effort than they deliver in practice. A migration thread introduces its own lock hierarchy, and its interaction with buffer pool locks can deadlock. Arbitrating bandwidth between migration and foreground queries tends to starve one side during bursts. Recovering from a crash in mid-migration needs an extra handling path, which complicates the redo/undo logic. Confining placement decisions to the two natural control points, eviction and flush, reuses the existing buffer manager control path and introduces no additional synchronization — so operators face a single, well-understood control loop.
Tiering must cooperate with the recovery and backup protocols. Data movement that runs independently of the recovery logic updates metadata without transactional context, and pages can end up duplicated or lost after a crash. In CloudJump III, a placement change affects only a page's physical location and is fully orthogonal to the logical version identified by its LSN. Correctness risks of this kind rarely show up in testing; they surface in production as serious failures.
From CloudJump I's I/O path redesign, to CloudJump II's MVD multi-version shared storage, to CloudJump III's engine-integrated tiering, the through-line is the same: build the storage foundation of a cloud-native database on standard cloud storage services, keep the complexity in the compute/engine layer, and keep the storage layer simple and general. That is the payoff of decoupling the database from the storage layer. Compared with Aurora's architecture, CloudJump can benefit fully from advances in the storage layer without building its own page-server layer to interface with different storage media.
CloudJump III shows that getting predictable performance out of tiered storage comes down to putting tiering into the same control loop as caching and recovery: place data accurately using what the engine can see, bound the cost of a miss with an eviction-centric mechanism, and guarantee zero-downtime backup correctness with a snapshot versioning protocol. The result: a small slice of fast-tier capacity buys near-local throughput, stable access latency, and markedly better cost performance.
For more, see the paper: "CloudJump III: Optimizing Cloud Databases for Tiered Storage".
Chen, Z., Sha, M., Li, F., Wang, S., Huang, B., Ma, G., … & Wang, Y. (2026). CloudJump III: Optimizing Cloud Databases for Tiered Storage. In Proceedings of the 2026 International Conference on Management of Data (SIGMOD).
Chen, Z., Yang, X., Sha, M., Li, F., Wang, K., Miao, Z., … & Wang, S. (2025, June). CloudJump II: Optimizing Cloud Databases for Shared Storage. In Companion of the 2025 International Conference on Management of Data (pp. 336-349).
Chen, Z., Yang, X., Li, F., Cheng, X., Hu, Q., Miao, Z., … & Wang, S. (2022). CloudJump: optimizing cloud databases for cloud storages. Proceedings of the VLDB Endowment, 15(12), 3432-3444.
Highlights | Alibaba Cloud Database New Features in September 2026
ApsaraDB - December 21, 2022
ApsaraDB - March 17, 2026
ApsaraDB - December 26, 2023
ApsaraDB - October 22, 2020
ApsaraDB - November 28, 2019
ApsaraDB - June 27, 2022
Database for FinTech Solution
Leverage cloud-native database solutions dedicated for FinTech.
Learn More
Database Migration Solution
Migrating to fully managed cloud databases brings a host of benefits including scalability, reliability, and cost efficiency.
Learn More
Oracle Database Migration Solution
Migrate your legacy Oracle databases to Alibaba Cloud to save on long-term costs and take advantage of improved scalability, reliability, robust security, high performance, and cloud-native features.
Learn More
Database Gateway
A tool product specially designed for remote access to private network databases
Learn MoreMore Posts by ApsaraDB
