Solution highlights: 3x higher throughput, one-tenth the write latency, and high elasticity for easy scaling during sales promotions.
Business challenges
The self-managed open source HBase cluster had insufficient performance and could not handle the real-time writing and computing of hundreds of thousands of events per second.
The self-managed open source HBase cluster had critical garbage collection (GC) issues. This caused poor request stability, frequent failures, and significant availability problems.
The continuous growth in data volume made storage costs a major concern.
The self-managed open source HBase cluster lacked an Operations and Maintenance (O&M) platform. Manual scaling was costly and prone to operational failures.
Solutions
High concurrency and throughput capacity: 1) Lindorm uses Group Commit write optimization to improve batch write performance by 3x. 2) Lindorm uses an LLC triplicate logging feature and a Quorum mechanism to reduce write latency by 50%. 3) Lindorm provides full linear scalability and natively supports tens of millions of reads and writes per second for a single table, eliminating the need for sharding.
Through GC optimization, Lindorm reduces P999 latency spikes by an order of magnitude. This makes application requests more stable.
Lindorm supports advanced compression algorithms that halve storage overhead and reduce storage costs.
The hot and cold data separation feature of a Lindorm cluster separates hot and cold data within a single table. This requires zero application changes and reduces storage costs.
Lindorm uses a storage-compute disaggregation architecture, which allows storage and compute nodes to be scaled independently as needed. These scaling operations are transparent to your application and require no code changes. Data and requests are automatically balanced, resulting in zero O&M.
Customer value
After migrating to Lindorm, the service achieved 3x the throughput capacity of the self-managed cluster. It now easily supports 200,000 writes per second with write latency reduced to one-tenth.
After migrating to Lindorm, compression efficiency doubled compared to the self-managed open source HBase, reducing storage costs by more than 50%. Using tiered storage of hot and cold data for a single table within the cluster can further reduce storage costs.
Lindorm uses Alibaba Dragonwell (AJDK) ZGC for GC optimization. Kernel-level optimizations eliminate request instability caused by GC. The system has been stable with no failures since the migration.
The Lindorm console provides one-click scaling to easily handle major sales promotions. This also significantly reduces O&M costs and the risk of unpredictable issues from manual operations.