Accelerate Elasticsearch queries with the FalconSeek cloud-native kernel - Elasticsearch

Elasticsearch (ES) can experience performance bottlenecks when handling complex aggregate queries, high-cardinality term queries, and vector searches. The FalconSeek cloud-native kernel is built on the proprietary Havenask engine from Alibaba. It uses C++ columnar memory and a fully asynchronous framework to significantly improve query performance while remaining fully compatible with the ES ecosystem. This topic describes how to enable FalconSeek to accelerate queries. It covers instance creation, index-level and field-level configurations, and query policy management. It also provides optimization practices for performance-sensitive scenarios, such as vector searches.

Scope

Invitational preview and free usage: FalconSeek is in a limited free testing phase. To be added to the whitelist, contact technical support.
Version restrictions: This feature requires Elasticsearch version 8.17.0. You cannot upgrade or downgrade the version.
Instance restrictions: You can enable FalconSeek only for newly purchased Enhanced Edition (Vector Search) instances. Existing instances are not supported.

Create an instance and enable FalconSeek

You can enable FalconSeek only when you create a new instance. Follow these steps to complete the configuration.

On the instance purchase page, enable FalconSeek.
- Cluster Type: Must be Vector Enhanced Edition.
- Instance Type: Select 8.17.0 (Recommended For Vector Scenarios).
- Advanced Enhanced Features: Select FalconSeek Cloud-native Kernel.
Set other parameters, such as VPC and Cluster Specification, as required, and then complete the payment process.
After the instance is created, run the following command in the Kibana console to confirm that FalconSeek is enabled at the cluster level.
```
GET _cluster/settings?include_defaults&filter_path=defaults.havenask.engine.enabled
```
If the enabled field in the response is true, the feature is enabled.
```
{
  "defaults": {
    "havenask": {
      "engine": {
        "enabled": "true"
      }
    }
  }
}
```

Use FalconSeek in an index

After you enable FalconSeek for an instance, it is not enabled for any index by default. You must use index settings to explicitly enable the FalconSeek engine for the specific indexes that you want to accelerate.

Enable FalconSeek for a new index

When you create an index, add the parameter "index.havenask.engine.enabled": true in the settings to enable FalconSeek.

PUT falcon_seek_test
{
  "settings": {
    "index.havenask.engine.enabled": true    // Enable FalconSeek
  },
  "mappings": {
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}

Write data.

POST falcon_seek_test/_bulk
{"index":{}}
{"foo":"hello"}
{"index":{}}
{"foo":"world"}
{"index":{}}
{"foo":"cpp"}
{"index":{}}
{"foo":"java"}

Query the data. FalconSeek is used by default.

GET falcon_seek_test/_search
{
  "query": {
    "term":{
      "foo":{
        "value": "hello"
      }
    }
  },
  "sort": [
    {
      "_doc": {
        "order": "desc"
      }
    }
  ]
}

Enable FalconSeek for an existing index

The following command creates an index but does not enable acceleration.

PUT my_existing_index
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "foo": {
        "type": "keyword"
      }
    }
  }
}

Use the _settings API to update the index configuration and dynamically enable the feature.
```
PUT my_existing_index/_settings
{
  "index.havenask.engine.enabled": true
}
```
For this setting to take effect, you must close the index by running POST /my_existing_index/_close and then reopen it by running POST /my_existing_index/_open.

Manage query execution policies

FalconSeek provides flexible query execution policies. You can control whether a request is processed by FalconSeek or falls back to the native Elasticsearch engine. This can be configured at the index level or for an individual query. To configure the policy, use the index.havenask.engine.search.type parameter at the index level or the havenask_search_type parameter at the query level. The query-level parameter has a higher priority.

Policy	Description	Recommended scenario
`native` (Default)	Queries are prioritized for execution by FalconSeek. If an unsupported syntax or feature is encountered, the request automatically falls back to the native Elasticsearch engine.	General production environments that require both performance and compatibility.
`es`	Queries are forced to run on the native Elasticsearch engine.	When you need to compare performance or bypass an unknown issue in FalconSeek.
`native-direct`	Queries are forced to run on the FalconSeek engine. If an unsupported syntax or feature is encountered, the system throws an exception directly and does not fall back.	Debug scenarios.

Set the default query policy at the index level

The following example sets all queries on my_index to run on the native Elasticsearch engine by default.

PUT my_index/_settings
{
  "index.havenask.engine.search.type": "es"
}

Temporarily override the default policy for a query

If the my_index index uses the native policy by default, you can force a specific query to use the native Elasticsearch engine by adding the havenask_search_type parameter to the query request.

Method 1: As a URL parameter

GET my_index/_search?havenask_search_type=es
{
  "query": {
    "match_all": {}
  }
}

Method 2: As a request body parameter

GET my_index/_search
{
  "query": {
    "match_all": {}
  },
  "havenask_search_type": "es"
}

Accelerate vector searches with FalconSeek

FalconSeek adds a new C++-based vector index on top of the original ES index schema. The FalconSeek vector index is developed by Alibaba and supports major services within Alibaba Group, such as search and recommendation on Taobao and Tmall, and Pailitao (search by image). You can use the high-performance vector index integrated into the FalconSeek kernel to build efficient AI applications, such as search by image and semantic search.

Create a vector index, define a dense_vector field named mage_vector in the mappings to store image or text vectors, and enable FalconSeek for the index.

PUT vector_index
{
  "settings": {
    "index.havenask.engine.enabled": true
  },
  "mappings": {
    "properties": {
      "image_vector": {
        "type": "dense_vector",
        "dims": 128 ,
        "index_options": {
          "type": "havenask_native"
        }
      },
      "title": {
        "type": "keyword"
      }
    }
  }
}

Write vector data. Use the _bulk or _index API to write documents. The value of the vector field is an array of floating-point numbers.

POST vector_index/_bulk
{"index":{"_id":"1"}}
{"image_vector":[0.12, -0.05, 0.08, 0.24, -0.17, 0.31, 0.02, -0.19, 0.11, 0.28,
    -0.03, 0.15, 0.22, -0.11, 0.09, 0.33, -0.07, 0.14, 0.26, -0.21,
    0.18, 0.29, -0.13, 0.06, 0.35, -0.08, 0.16, 0.23, -0.15, 0.12, 
    0.27, -0.22, 0.19, 0.32, -0.14, 0.07, 0.25, -0.18, 0.13, 0.30,
    -0.09, 0.17, 0.24, -0.16, 0.10, 0.34, -0.10, 0.20, 0.31, -0.23,
    0.15, 0.28, -0.12, 0.11, 0.26, -0.19, 0.14, 0.29, -0.17, 0.08,
    0.22, -0.20, 0.16, 0.27, -0.15, 0.09, 0.25, -0.21, 0.18, 0.30,
    -0.13, 0.07, 0.24, -0.22, 0.19, 0.32, -0.16, 0.10, 0.26, -0.18,
    0.12, 0.28, -0.14, 0.06, 0.23, -0.19, 0.15, 0.29, -0.11, 0.05,
    0.21, -0.17, 0.13, 0.27, -0.10, 0.04, 0.20, -0.15, 0.11, 0.25,
    -0.09, 0.03, 0.19, -0.13, 0.10, 0.24, -0.08, 0.02, 0.18, -0.12,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22, 0.05, -0.06,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22],"title":"Image One"}
{"index":{"_id":"2"}}
{"image_vector":[0.12, -0.05, 0.08, 0.24, -0.17, 0.31, 0.02, -0.19, 0.11, 0.28,
    -0.03, 0.15, 0.22, -0.11, 0.09, 0.33, -0.07, 0.14, 0.26, -0.21,
    0.18, 0.29, -0.13, 0.06, 0.35, -0.08, 0.16, 0.23, -0.15, 0.12, 
    0.27, -0.22, 0.19, 0.32, -0.14, 0.07, 0.25, -0.18, 0.13, 0.30,
    -0.09, 0.17, 0.24, -0.16, 0.10, 0.34, -0.10, 0.20, 0.31, -0.23,
    0.15, 0.28, -0.12, 0.11, 0.26, -0.19, 0.14, 0.29, -0.17, 0.08,
    0.22, -0.20, 0.16, 0.27, -0.15, 0.09, 0.25, -0.21, 0.18, 0.30,
    -0.13, 0.07, 0.24, -0.22, 0.19, 0.32, -0.16, 0.10, 0.26, -0.18,
    0.12, 0.28, -0.14, 0.06, 0.23, -0.19, 0.15, 0.29, -0.11, 0.05,
    0.21, -0.17, 0.13, 0.27, -0.10, 0.04, 0.20, -0.15, 0.11, 0.25,
    -0.09, 0.03, 0.19, -0.13, 0.10, 0.24, -0.08, 0.02, 0.18, -0.12,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22, 0.05, -0.06,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22],"title":"Image Two"}

Perform a k-NN vector search. Use the knn query clause to find the top K results that are most similar to a given vector.

 GET vector_index/_search
{
  "knn": {
    "field": "image_vector",
    "query_vector": [0.12, -0.05, 0.08, 0.24, -0.17, 0.31, 0.02, -0.19, 0.11, 0.28,
    -0.03, 0.15, 0.22, -0.11, 0.09, 0.33, -0.07, 0.14, 0.26, -0.21,
    0.18, 0.29, -0.13, 0.06, 0.35, -0.08, 0.16, 0.23, -0.15, 0.12, 
    0.27, -0.22, 0.19, 0.32, -0.14, 0.07, 0.25, -0.18, 0.13, 0.30,
    -0.09, 0.17, 0.24, -0.16, 0.10, 0.34, -0.10, 0.20, 0.31, -0.23,
    0.15, 0.28, -0.12, 0.11, 0.26, -0.19, 0.14, 0.29, -0.17, 0.08,
    0.22, -0.20, 0.16, 0.27, -0.15, 0.09, 0.25, -0.21, 0.18, 0.30,
    -0.13, 0.07, 0.24, -0.22, 0.19, 0.32, -0.16, 0.10, 0.26, -0.18,
    0.12, 0.28, -0.14, 0.06, 0.23, -0.19, 0.15, 0.29, -0.11, 0.05,
    0.21, -0.17, 0.13, 0.27, -0.10, 0.04, 0.20, -0.15, 0.11, 0.25,
    -0.09, 0.03, 0.19, -0.13, 0.10, 0.24, -0.08, 0.02, 0.18, -0.12,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, 0.08, 0.22, 0.05, -0.06,
    0.09, 0.23, -0.07, 0.01, 0.17, -0.11, -0.08,-0.22],
    "k": 10,
    "num_candidates": 100
  },
  "fields": ["title"]
}

For more information about using the FalconSeek vector index, see the FalconSeek Vector Index User Guide.

How it works

The FalconSeek cloud-native kernel replaces some of the underlying Lucene components of Elasticsearch with a highly optimized C++ engine. When you send a query request, the request first reaches the FalconSeek plugin layer. This layer determines whether the query can be efficiently processed by the FalconSeek engine:

Supported queries: The request is forwarded to the high-performance FalconSeek engine for execution.
Unsupported queries: Based on your configured policy, the system automatically falls back to the native Elasticsearch engine or returns an error message.

This mechanism ensures high performance while maintaining compatibility and stability.

FAQ

Q: Can I enable FalconSeek on an existing Elasticsearch instance?

A: No, you cannot. FalconSeek can be enabled only on newly purchased instances. To use this feature with an existing instance, you must create a new, compatible instance and migrate your data using a tool such as Alibaba Cloud Data Transmission Service (DTS) or Logstash.

Q: What happens if I run a query that FalconSeek does not support?

A: The outcome depends on your query execution policy. With the default native policy, the query automatically falls back to the native Elasticsearch engine. You still receive the correct result, but without the performance acceleration provided by FalconSeek. With the native-direct policy, the system returns an error message.