HBase is a flexible database, but its performance depends on proper configuration and usage. This topic describes several methods for optimizing read performance. In production environments, you might encounter issues such as Full GC, out-of-memory (OOM) errors, Region-In-Transition (RIT) problems, and high read latency. While better hardware can mitigate these issues, proper optimization is essential for the best performance.
Optimizations can be divided into the following categories:
client-side optimization, server-side optimization, and platform optimization (ApsaraDB for HBase)
Client-side optimization
Batch get requests
Batch requests substantially reduce RPC calls between the client and server, which significantly improves throughput.
Result[] re= table.get(List<Get> gets)Set cache size for large scans
When scanning a large amount of data, the client sends a single request, and the server returns the data in batches. This design prevents a single, massive data transfer from overwhelming the client or server. By default, data is loaded into a local cache that holds 100 rows. For large scans, this can result in hundreds or even tens of thousands of RPC requests. Increase the cache size for these operations.
scan.setCaching(int caching) // For large scans, you can set this to 1000.Specify column families or qualifiers
HBase is a column-family-oriented database. Data within the same column family is stored together, separate from other families. To reduce I/O, always specify the required column families or specific column qualifiers in your request.
Disable cache for offline tasks
When accessing HBase for offline computing, data is typically read only once. To avoid filling the blockcache with data that will not be reused, disable the blockcache for these scans.
scan.setBlockCache(false)Server-side optimization
Ensure request load is balanced
Check if the read load is concentrated on one or a few servers, especially during peak hours. You can monitor this through the HBase UI on the HBase Management Platform. If you identify a significant hotspot, the best long-term solution is to redesign your rowkey. As a short-term fix, consider splitting the hot region.
Configure blockcache size
The blockcache is a read cache critical to read performance. For read-heavy workloads, use instances with a 1:4 vCPU-to-memory ratio, such as 8 vCPU/32 GiB or 16 vCPU/64 GiB. You can improve read performance by increasing the blockcache size and decreasing the Memstore size.
In the ApsaraDB for HBase console, set hfile.block.cache.size to 0.5 and hbase.regionserver.global.memstore.size to 0.3, and then restart the service.
Also, change hbase.regionserver.global.memstore.lowerLimit to 0.24.
Manage HFile count
Read operations require opening HFiles. A larger number of HFiles leads to more I/O operations and higher read latency. To manage this, run major compaction periodically. If your workload is light at night, schedule the major compaction to run during off-peak hours.
Monitor resource consumption from compactions
Compaction merges smaller HFiles into larger ones, which improves subsequent read performance but also consumes significant system resources. A minor compaction typically does not cause heavy resource usage unless it is misconfigured. Avoid running a major compaction during peak hours. Schedule major compactions for off-peak periods.
Use bloom filters correctly
A bloom filter helps determine if an HFile might contain a specific row, which avoids unnecessary I/O operations and improves read performance. When creating a table, enable a bloom filter on the rowkey by setting BLOOMFILTER => 'ROW'.
Platform optimization
Improve data locality
If an HFile is stored on the same node that serves its data, HBase can use Short-Circuit Local Read for faster access. The platform is optimized to maintain high data locality. After events like restarts or disk expansions, it automatically restores regions to their local nodes. Regular major compactions also help improve data locality.
Short-Circuit Local Read (enabled by default)
Normally, HDFS read operations go through a DataNode. With Short-Circuit Local Read enabled, the client can bypass the DataNode and read data directly from the local disk.
Hedged Read (enabled by default)
The system first attempts a local read using Short-Circuit Local Read. However, local reads can sometimes slow down due to transient disk or network issues. Hedged Read mitigates this problem. When a client initiates a local read, it starts a short timer. If the read does not complete before the timer expires, the client sends a second read request to a different DataNode. The client uses the first response it receives and discards the other request.
Disable swap space (disabled by default)
Swap space is a portion of a hard drive used as virtual memory when physical memory is insufficient. Using swap space introduces significant latency, so it is disabled by default on the ApsaraDB for HBase platform. However, disabling swap space can cause anon-rss to become high, and page reclaim may fail to free enough pages, which can cause the kernel to hang. The platform uses isolation measures to prevent this.