OpenStore intelligent hybrid storage engine - Elasticsearch

OpenStore is an elastic, efficient, and cost-effective log storage engine developed by the Alibaba Cloud Elasticsearch team for log analytics scenarios. It provides serverless storage with pay-as-you-go billing based on actual data usage, eliminating the need to provision storage capacity in advance. OpenStore moves beyond traditional hot-cold data separation, simplifying data ingestion and further reducing storage costs for large-scale cloud data.

Important

This feature is available in specific regions, including China (Hong Kong).

Background

In log analytics and observability scenarios, business or regulatory requirements often demand long-term data retention for archival and auditing. With open source Elasticsearch, this typically involves hot-cold data separation. Data older than 30 days is often moved to other storage media, such as OSS, by using cluster snapshots. While this approach works for long-term archiving, you cannot directly query archived data. To query it, you must first restore the snapshot to a cluster and wait for the indexes to initialize, which is complex and increases long-term storage costs.

OpenStore is a key feature of Alibaba Cloud Elasticsearch 7.10 Kernel-enhanced Edition. Combined with the managed Indexing Service, it supports low-cost, high-concurrency writes and long-term data retention for log analytics. You can create a 7.10 Kernel-enhanced Edition cluster on-demand and enable the OpenStore feature.

Note

You can check whether OpenStore is enabled and view its storage information in the Node Visualization section on the Basic Information page of your cluster. For more information, see View cluster status and node information.
For a cluster with OpenStore and Indexing Service enabled, the underlying storage service ensures high data availability. As a result, the Auto Snapshot feature is not supported.

Benefits

Massive storage: OpenStore uses a serverless storage model with pay-as-you-go billing, eliminating the need for upfront capacity planning. You are billed hourly based on actual usage, achieving 100% storage utilization.
Low cost: Data can be modified and updated in real time. Automated data tiering eliminates complex index lifecycle configurations, simplifying initial setup. Storage costs are up to 60% lower than local SATA disks and 70% lower than ultra disks.
High availability: A compute-storage separation architecture allows multiple replicas to share a single data copy at no extra cost. The underlying storage service ensures high data availability and provides durability of up to 99.9999999999% (twelve nines).
Improved query performance: For typical log analytics query and analysis tasks, performance improves by 100% compared to local SATA disks and is comparable to ultra disks or PL0 ESSDs.

Limitations

The following limitations apply when you purchase and use OpenStore.

Category	Description
Region	OpenStore is available only in the following regions. The actual regions available are displayed on the purchase page. China (Shanghai), China (Zhangjiakou), China (Hangzhou), China (Shenzhen), and China (Beijing) Singapore, Germany (Frankfurt), US (Virginia), and Japan (Tokyo)
Cluster version	Only Elasticsearch 7.10 Kernel-enhanced Edition clusters support OpenStore.
Cluster specifications	Only the following storage-optimized specifications are supported for OpenStore: 8-core 64 GB and 16-core 64 GB.
Cluster storage capacity	The maximum storage capacity per node is 30 TB. Note If you require a higher storage capacity per node, you can submit a ticket to apply for up to 50 TB.
Number of shard replicas	When you enable OpenStore, you must configure at least one replica per shard. Warning Multiple replicas share a single copy of data at no extra storage cost. Replicas ensure the reliability of write acceleration for local storage. If you do not configure replicas, recently written data may be lost and cannot be recovered.
Index template	For clusters with kernel version 1.8 or earlier, you must manually specify the `component-hybrid-openstore-index-template` composable template. For clusters with kernel version 1.9 or later, you do not need to manually specify the composable template. Note To manually delete an OpenStore index, you must delete both the index and its corresponding alias.
Index lifecycle configuration	Customizing the `freeze` action in the index lifecycle is not supported.
Query limits	The following aggregation query types are supported only on kernel version 1.9 or later. They are not supported on earlier kernel versions. PercentilesAggregation, PercentileRanksAggregation, SamplerAggregation, DiversifiedAggregation, SignificantTextAggregation, GeoDistanceAggregation, GeoHashGridAggregation, GeoTileGridAggregation, GeoBoundsAggregation, GeoCentroidAggregation, ScriptedMetricAggregation. The following aggregation query types are not supported on any kernel version. FiltersAggregation, AdjacencyMatrixAggregation, ReverseNestedAggregation, CompositeAggregation.
Cluster shard limit	Fewer than 80,000 shards per cluster is recommended.
Node shard limit	Fewer than 3,000 shards per node is recommended.
Size per shard	Less than 40 GB per shard is recommended.
Data disk write throughput	300 MB/s if the data disk usage is below 85%. 100 MB/s if the data disk usage is 85% or higher.

Use cases

OpenStore is ideal for scenarios that require writing and storing large volumes of data long-term, such as log retrieval and metrics analysis. It is best suited for workloads with low query QPS and higher tolerance for query latency.

The intelligent hybrid storage engine is also suitable for business scenarios that require real-time data updates without strict hot-cold data separation.

Hybrid storage architecture

混合存储架构

This architecture provides the following benefits:

Compute-storage separation: Unlike traditional hot-cold data separation architectures, this design further decouples compute and storage resources. You no longer need to manage storage capacity. This architecture provides elastic, pay-as-you-go storage and leverages cloud-native principles to improve cluster scalability. It significantly accelerates index migration and recovery, making it ideal for large-scale data scenarios.
Ease of use: With fully automated index lifecycle management, you only need to configure simple lifecycle policies. The engine manages hot-cold data separation and migrates data to OpenStore automatically.
Data consistency: OpenStore uses a Raft-based consensus protocol to ensure data consistency across storage media. It automatically handles data tiering and cache acceleration without user intervention and supports real-time data updates.

Performance testing

Test environment
- Dataset: A dataset from a typical log analytics scenario.
- Cluster specifications: The clusters used for comparison share the same configuration, which is optimized for log analytics.
  - Number of nodes: 10
  - Number of shards: 108
- Query conditions:
  - Query type: sort
  - Number of documents: 3,800,000,000

Test results

Storage type	Query time
Local SATA disk	Over 30s
ultra disk	12.229s
OpenStore	15.841s

Test conclusion:

With identical cluster configurations, query latency for log data on OpenStore is significantly lower than on local SATA disks and comparable to ultra disks. OpenStore is about 60% cheaper per unit than ultra disks and uses pay-as-you-go billing, eliminating the need for upfront capacity purchases.

Related metrics

Metric	Value
Access latency (local cache hit)	0.2 ms
Access latency (local cache miss)	50 ms to 400 ms
Access throughput (local cache hit)	1 GB/s
Access throughput (local cache miss)	750 MB/s
Use cases	Infrequently accessed data, such as monitoring logs, historical orders, and archived data