OpenStore Storage Engine (Log Analytics) is a serverless log storage engine developed by the Alibaba Cloud Elasticsearch team for log scenarios. It supports the pay-as-you-go billing method based on the actual volume of data that is stored, so you do not need to reserve cluster storage capacity in advance. OpenStore goes beyond the traditional hot-cold data separation architecture, which reduces the complexity of cluster data ingestion and lowers the cost of storing massive amounts of data in the cloud.
OpenStore Storage Engine (Log Analytics) is currently available in the China (Hong Kong) region. Support for other regions is coming soon.
Background information
In log observability scenarios, business or regulatory requirements often call for long-term data retention or archiving for audits. With open source Elasticsearch, you must separate hot and cold data in the cluster. You must also store data that is 30 days old or older on other storage media, such as Object Storage Service (OSS), by using cluster snapshots. This approach achieves long-term log data archiving, but you cannot query the stored data directly. Before you query the data, you must call the relevant API operations to restore the snapshot to the cluster and wait for the indexes in the snapshot to complete initialization. As a result, queries are complex and long-term storage costs are high.
OpenStore is an important feature of Alibaba Cloud Elasticsearch 7.10 Kernel-enhanced Edition. Combined with the managed Indexing Service, OpenStore delivers low-cost, high-concurrency write operations and long-term data retention for log scenarios. You can create a 7.10 Kernel-enhanced Edition cluster on demand and enable OpenStore Storage Engine (Log Analytics).
In the Node Visualization section of the Basic Information page of your cluster, you can check whether OpenStore is enabled, enable OpenStore, and view OpenStore storage information. For more information, see View cluster status and node information.
If your cluster has OpenStore and Indexing Service enabled, the underlying storage service ensures high data availability for the cluster. Therefore, you cannot configure Auto Snapshot.
Benefits
Massive storage — OpenStore uses serverless storage billing. You do not need to plan for or purchase storage capacity in advance. Actual data storage usage is metered by the hour, and storage resource utilization reaches 100%.
Low cost — Data can be modified in real time and supports updates on write. You do not need to configure complex index lifecycles, and data tiering completes automatically, which keeps the learning curve low. The unit price of data storage is 60% lower than that of local SATA disks and 70% lower than that of ultra disks.
High availability (HA) — With the storage-compute disaggregation architecture, multiple replicas share one copy of data at no additional storage cost. The underlying storage service ensures high data availability for the cluster and provides data durability of up to 99.9999999999% (twelve nines).
Improved query performance — For common query and analysis tasks in typical log scenarios, performance improves by 100% compared with local SATA disks and is comparable to ultra disks or PL0 ESSDs.
Limits
The following limits apply when you purchase and use OpenStore storage.
Category | Description |
Region | OpenStore is available only in the following regions. The actual available regions are displayed in the console:
|
Cluster version | 9.4.0, 8.17.0, 7.10. |
Cluster specifications | Only the 8-core 64 GB and 16-core 64 GB OpenStore storage specifications are supported. |
Cluster storage capacity | The maximum data storage capacity per node is 30 TB. Note If you require a larger storage capacity per node, submit a ticket to apply. A maximum of 50 TB is supported. |
Number of shard replicas | When you enable OpenStore Storage Engine (Log Analytics), the number of shard replicas must be greater than or equal to 1. Warning Multiple replicas share one copy of data at no additional storage cost, and they ensure the reliability of local storage write acceleration. If you do not configure multiple replicas, some real-time written data may be lost and cannot be recovered. |
Index template |
Note To manually delete an OpenStore index, you must delete the index together with its corresponding alias. Otherwise, the deletion fails. |
Index lifecycle configuration | You cannot customize freeze in an index lifecycle. |
Query limits |
|
Cluster shard limit | (Recommended) Fewer than 80,000 shards per cluster. |
Node shard limit | (Recommended) Fewer than 3,000 shards per node. |
Size per shard | (Recommended) Less than 40 GB per shard. |
Data disk write throughput | 300 MB/s when the actual data disk usage is below 85%; 100 MB/s when the actual data disk usage is 85% or higher. |
Scenarios
OpenStore Storage Engine (Log Analytics) is developed by Alibaba Cloud Elasticsearch. It suits scenarios such as log retrieval and metrics analysis that require massive amounts of data to be written and stored over the long term. These scenarios also have low query QPS and a relatively high tolerance for query latency.
OpenStore is also suitable for business scenarios that require real-time data updates and have no strict hot-cold data separation.
Hybrid storage architecture
OpenStore Storage Engine (Log Analytics) uses a hybrid storage architecture with storage-compute disaggregation. Its architecture design principles and core benefits, such as storage-compute disaggregation, data consistency, ease of use, and high availability, are the same as those of the OpenStore storage engine. For more information, see the OpenStore storage engine topic in the same directory.
Performance test
Test environment
Dataset: A dataset from a log scenario.
Cluster specifications: All clusters use the same configuration for a log scenario, where:
Number of nodes: 10
Number of shards: 108
Query conditions:
Query type: sort
Number of documents: 3,800,000,000
Test results
Storage type
Query time
Local SATA disk
Over 30 seconds
Ultra disk
12.229 seconds
OpenStore storage
15.841 seconds
Test conclusion:
With the same cluster configuration, querying log data stored in OpenStore takes significantly less time than querying data stored on local SATA disks. The query time is roughly the same as that for data stored on ultra disks. The unit price of OpenStore storage is about 60% lower than that of ultra disk storage. OpenStore also uses the pay-as-you-go billing method, so you do not need to purchase storage capacity in advance. Therefore, OpenStore storage can reduce your costs.