analytic-search is a log query plug-in that is developed by the Alibaba Cloud Elasticsearch team. The analytic-search plug-in supports the time field index.sort that is used to accelerate queries based on specific search conditions on the Discover page in the Kibana console of an Elasticsearch cluster. The analytic-search plug-in also provides the concurrent query feature. This plug-in helps significantly reduce the period of time that is required to query data. This topic describes how to use the analytic-search plug-in.
Background information
The following descriptions provide the features supported by the analytic-search plug-in, the scenarios in which the features can be used, and the information about the performance tests performed on the features.
Acceleration feature for queries performed on the Discover page of the Kibana console
- Use scenarios: This feature is suitable for log query scenarios. For example, you can use the analytic-search plug-in to accelerate unconditional queries and single-condition queries on the Discover page of the Kibana console.
- Benefits: Index merging policies and date histogram aggregation policies are optimized. This significantly improves the performance of unconditional queries and single-condition queries in log query scenarios. In scenarios in which more than 1 TB of data is added each day, the period of time that is required to complete a query is reduced from minutes to 5 seconds or less.
- Performance test:
- Test environment
- Node: 10 nodes, each of which offers 16 vCPUs and 64 GiB of memory.
- Dataset: business log data of 60 billion documents each day. The data is stored in 12 indexes, each of which is configured with 60 shards.
- The following table provides the percentages by which the period of time required
to query data from different storage media is reduced after the acceleration feature
is enabled for queries performed on the Discover page of the Kibana console.
Query type Standard SSD Ultra disk OpenStore Unconditional query Reduced by 96% Reduced by 95% Reduced by 94% Single-condition query Reduced by 88% Reduced by 77% Reduced by 85% Multi-condition query Reduced by 8% Reduced by 11% Reduced by 14%
- Test environment
Concurrent query feature
- Use scenarios: This feature is suitable for scenarios in which the queries per second (QPS) is low, a long period of time is required to return results for queries, and computing resources of nodes are sufficient.
- Benefits: Concurrent threads can be used to return results for queries, and the average period of time required to return results for queries is reduced by 50%. Resource utilization is improved.
- Performance test:
- Test environment
- Node: three warm-hot shared computing nodes provided by OpenStore, each of which offers
16 vCPUs and 64 GiB of memory.
Note Warm-hot shared computing nodes provided by OpenStore are available for purchase in the Elasticsearch console only at the Alibaba Cloud China site (aliyun.com).
- Dataset: 1.6 TB of business log data. The data is stored as 6 billion documents in an index for which 60 shards are configured.
- Query: 3 TermQuery(and) + TimeRange + Sort + Datehistogram. 10 million documents can be queried from a single shard at a hit rate of 10%.
- Node: three warm-hot shared computing nodes provided by OpenStore, each of which offers
16 vCPUs and 64 GiB of memory.
- Test results:
- Period of time required to query data from a single shard is reduced by 65%.
- Period of time required to query data from multiple shards is reduced by 53%.
- Test environment
Prerequisites
Enable the acceleration feature for queries performed on the Discover page of the Kibana console
{
"settings": {
"index.points.same_sort_order_as_index_sort": true,
"index.sort.field": [
"@timestamp"
],
"index.sort.order": [
"desc"
]
},
"mappings": {
"properties": {
"@timestamp": {
"type": "date"
}
}
}
}