Boost Elasticsearch Query Performance with FalconSeek - Elasticsearch - Alibaba Cloud - Elasticsearch

Elasticsearch may face performance bottlenecks when processing complex aggregate queries, high-cardinality term queries, and vector search. The FalconSeek cloud-native kernel, built on Alibaba's proprietary Havenask engine, uses C++ columnar memory and a fully asynchronous framework to significantly improve query performance while maintaining high compatibility with the Elasticsearch ecosystem. This topic describes how to enable FalconSeek to accelerate queries by default, covering instance creation, index-level and field-level configuration, query strategy management, and optimization practices for performance-sensitive scenarios such as vector search.

Applicable scope

Invitation-only free preview: FalconSeek is currently in a limited free preview phase.
Version requirement: This feature requires Elasticsearch version 8.17.0 or 9.4.0.
New instances: You can directly enable FalconSeek when you purchase a version 8.17.0 or 9.4.0 instance.
Existing instances: For existing version 8.17.0 or 9.4.0 instances, you can log on to the console and go to the Basic Information page of the instance. In the Engine Kernel section, click Configuration Change to enable or disable FalconSeek.

Create an instance and enable FalconSeek

When you purchase a version 8.17.0 or 9.4.0 instance, you can directly enable FalconSeek on the purchase page. To enable FalconSeek on an existing version 8.17.0 or 9.4.0 instance, see Applicable scope above. To create a new instance with FalconSeek, follow these steps.

On the instance purchase page, enable FalconSeek.
Elasticsearch Version: Select 8.17.0 or 9.4.0.
Advanced Enhanced Features: Select FalconSeek Cloud-Native Kernel.
Configure other parameters based on your business needs, such as VPC and Cluster Specification, and then complete the payment.
After the instance is created, run the following command in the Kibana console to confirm that FalconSeek has been enabled at the cluster level.
```
GET _cluster/settings?include_defaults&filter_path=defaults.havenask.engine.enabled
```
If the enabled field in the response is true, the feature has been successfully enabled.
```
{
  "defaults": {
    "havenask": {
      "engine": {
        "enabled": "true"
      }
    }
  }
}
```

Use FalconSeek in an index

FalconSeek is not enabled on any index by default. You must explicitly enable the engine for each index you want to accelerate.

Enable FalconSeek for a new index

When you create an index, add "index.havenask.engine.enabled": true to settings to enable FalconSeek.

PUT falcon_seek_test
{
  "settings": {
    "index.havenask.engine.enabled": true    // Enable the FalconSeek feature
  },
  "mappings": {
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}

Ingest data.

POST falcon_seek_test/_bulk
{"index":{}}
{"foo":"hello"}
{"index":{}}
{"foo":"world"}
{"index":{}}
{"foo":"cpp"}
{"index":{}}
{"foo":"java"}

Query the data. By default, the query uses FalconSeek.

GET falcon_seek_test/_search
{
  "query": {
    "term":{
      "foo":{
        "value": "hello"
      }
    }
  },
  "sort": [
    {
      "_doc": {
        "order": "desc"
      }
    }
  ]
}

Enable FalconSeek for an existing index

The following command creates an index but does not enable acceleration.

PUT my_existing_index
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}

Use the _settings API to update the index configuration to dynamically enable FalconSeek.
```
PUT my_existing_index/_settings
{
  "index.havenask.engine.enabled": true
}
```
For FalconSeek to take effect in this scenario, you must first close the index by running POST /my_existing_index/_close and then reopen it by running POST /my_existing_index/_open.

Manage query execution strategies

FalconSeek provides flexible query execution strategies to control whether a request is processed by FalconSeek or falls back to the Elasticsearch native engine. Configure this at the index level with index.havenask.engine.search.type or per query with havenask_search_type. The query-level parameter takes precedence.

Strategy	Description	Recommended use case
`native` (default)	Queries are first processed by FalconSeek. If an unsupported syntax or feature is encountered, the request automatically falls back to the Elasticsearch native engine.	General production workloads that require both performance and compatibility.
`es`	Forces queries to be executed by the Elasticsearch native engine.	For performance comparisons or to bypass unknown issues in FalconSeek.
`native-direct`	Forces the FalconSeek engine to execute queries. If an unsupported syntax or feature is encountered, the system returns an error instead of falling back.	Debugging.

Set the index-level query strategy

The following example configures all queries on my_index to use the Elasticsearch native engine by default.

PUT my_index/_settings
{
  "index.havenask.engine.search.type": "es"
}

Override the strategy per query

If my_index uses the native strategy by default, you can force a specific query to use the Elasticsearch native engine by adding the havenask_search_type parameter to the request.

Method 1: As a URL parameter

GET my_index/_search?havenask_search_type=es
{
  "query": {
    "match_all": {}
  }
}

Method 2: As a request body parameter

GET my_index/_search
{
  "query": {
    "match_all": {}
  },
  "havenask_search_type": "es"
}

Use FalconSeek to accelerate vector search

FalconSeek adds a C++-based vector index on top of the existing Elasticsearch index structure. This proprietary vector index, developed by Alibaba, powers core services across Alibaba Group, including search, recommendations, and reverse image search on Taobao and Tmall. Use this high-performance vector index to build efficient AI applications such as reverse image search and semantic search.

Create a vector index. In the mappings, define a field of type dense_vector named image_vector to store image or text vectors. Ensure that FalconSeek is enabled for this index.

PUT vector_index
{
  "settings": {
    "index.havenask.engine.enabled": true
  },
  "mappings": {
    "properties": {
      "image_vector": {
        "type": "dense_vector",
        "dims": 128 ,
        "index_options": {
          "type": "havenask_native"
        }
      },
      "title": {
        "type": "keyword"
      }
    }
  }
}

Ingest vector data. Use the _bulk or _index API to write documents. The vector field value must be an array of floating-point numbers.

POST vector_index/_bulk
{"index":{"_id":"1"}}
{"image_vector":[0.12, -0.05, 0.08, 0.24, -0.17, 0.31, 0.02, -0.19, 0.11, 0.28,
    -0.03, 0.15, 0.22, -0.11, 0.09, 0.33, -0.07, 0.14, 0.26, -0.21,
    0.18, 0.29, -0.13, 0.06, 0.35, -0.08, 0.16, 0.23, -0.15, 0.12, 
    0.27, -0.22, 0.19, 0.32, -0.14, 0.07, 0.25, -0.18, 0.13, 0.30,
    -0.09, 0.17, 0.24, -0.16, 0.10, 0.34, -0.10, 0.20, 0.31, -0.23,
    0.15, 0.28, -0.12, 0.11, 0.26, -0.19, 0.14, 0.29, -0.17, 0.08,
    0.22, -0.20, 0.16, 0.27, -0.15, 0.09, 0.25, -0.21, 0.18, 0.30,
    -0.13, 0.07, 0.24, -0.22, 0.19, 0.32, -0.16, 0.10, 0.26, -0.18,
    0.12, 0.28, -0.14, 0.06, 0.23, -0.19, 0.15, 0.29, -0.11, 0.05,
    0.21, -0.17, 0.13, 0.27, -0.10, 0.04, 0.20, -0.15, 0.11, 0.25,
    -0.09, 0.03, 0.19, -0.13, 0.10, 0.24, -0.08, 0.02, 0.18, -0.12,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22, 0.05, -0.06,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22],"title":"Image One"}
{"index":{"_id":"2"}}
{"image_vector":[0.12, -0.05, 0.08, 0.24, -0.17, 0.31, 0.02, -0.19, 0.11, 0.28,
    -0.03, 0.15, 0.22, -0.11, 0.09, 0.33, -0.07, 0.14, 0.26, -0.21,
    0.18, 0.29, -0.13, 0.06, 0.35, -0.08, 0.16, 0.23, -0.15, 0.12, 
    0.27, -0.22, 0.19, 0.32, -0.14, 0.07, 0.25, -0.18, 0.13, 0.30,
    -0.09, 0.17, 0.24, -0.16, 0.10, 0.34, -0.10, 0.20, 0.31, -0.23,
    0.15, 0.28, -0.12, 0.11, 0.26, -0.19, 0.14, 0.29, -0.17, 0.08,
    0.22, -0.20, 0.16, 0.27, -0.15, 0.09, 0.25, -0.21, 0.18, 0.30,
    -0.13, 0.07, 0.24, -0.22, 0.19, 0.32, -0.16, 0.10, 0.26, -0.18,
    0.12, 0.28, -0.14, 0.06, 0.23, -0.19, 0.15, 0.29, -0.11, 0.05,
    0.21, -0.17, 0.13, 0.27, -0.10, 0.04, 0.20, -0.15, 0.11, 0.25,
    -0.09, 0.03, 0.19, -0.13, 0.10, 0.24, -0.08, 0.02, 0.18, -0.12,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22, 0.05, -0.06,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22],"title":"Image Two"}

Perform a k-NN vector search. Use the knn query clause to find the top K results that are most similar to a given vector.

 GET vector_index/_search
{
  "knn": {
    "field": "image_vector",
    "query_vector": [0.12, -0.05, 0.08, 0.24, -0.17, 0.31, 0.02, -0.19, 0.11, 0.28,
    -0.03, 0.15, 0.22, -0.11, 0.09, 0.33, -0.07, 0.14, 0.26, -0.21,
    0.18, 0.29, -0.13, 0.06, 0.35, -0.08, 0.16, 0.23, -0.15, 0.12, 
    0.27, -0.22, 0.19, 0.32, -0.14, 0.07, 0.25, -0.18, 0.13, 0.30,
    -0.09, 0.17, 0.24, -0.16, 0.10, 0.34, -0.10, 0.20, 0.31, -0.23,
    0.15, 0.28, -0.12, 0.11, 0.26, -0.19, 0.14, 0.29, -0.17, 0.08,
    0.22, -0.20, 0.16, 0.27, -0.15, 0.09, 0.25, -0.21, 0.18, 0.30,
    -0.13, 0.07, 0.24, -0.22, 0.19, 0.32, -0.16, 0.10, 0.26, -0.18,
    0.12, 0.28, -0.14, 0.06, 0.23, -0.19, 0.15, 0.29, -0.11, 0.05,
    0.21, -0.17, 0.13, 0.27, -0.10, 0.04, 0.20, -0.15, 0.11, 0.25,
    -0.09, 0.03, 0.19, -0.13, 0.10, 0.24, -0.08, 0.02, 0.18, -0.12,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22, 0.05, -0.06,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, -0.08,-0.22],
    "k": 10,
    "num_candidates": 100
  },
  "fields": ["title"]
}

For more information about using the FalconSeek vector index, see FalconSeek vector index user guide.

How it works

The FalconSeek cloud-native kernel replaces parts of the underlying Lucene components in Elasticsearch with a highly optimized C++ engine. When you send a query, the FalconSeek plugin layer determines whether the FalconSeek engine can process it:

Supported queries: The request is forwarded to the FalconSeek engine for execution.
Unsupported queries: Depending on your configured strategy, the query automatically falls back to the Elasticsearch native engine, or the system returns an error.

This mechanism ensures high performance while maintaining compatibility.

FAQ

Q: Can I enable FalconSeek on an existing Elasticsearch instance?

A: It depends. For version 8.17.0 or 9.4.0 instances (including existing instances), you can enable FalconSeek in the Engine Kernel section on the Basic Information page by clicking Configuration Change. If your existing instance is not version 8.17.0 or 9.4.0, you cannot directly enable FalconSeek. You need to purchase a new version 8.17.0 or 9.4.0 instance and use Alibaba Cloud Data Transmission Service (DTS) or Logstash to migrate your data to the new instance. For more information, see Migration method selection.

Q: What happens if I run a query that FalconSeek does not support?

A: It depends on your query execution strategy. In the default native mode, the query automatically falls back to the Elasticsearch native engine. You will still get the correct result, but without the performance acceleration from FalconSeek. In native-direct mode, the system will return an error.