Elasticsearch:Guide to the Alibaba Cloud Elasticsearch vector engine

1. Create an index

First, create an index suitable for storing vector data. The following code provides an example of an index definition:

PUT /my_vector_index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 3
      },
      "my_text" : {
        "type" : "keyword"
      }
    }
  }
}

2. Import data

You can import data into the Elasticsearch vector index using various methods. For example, you can use the Bulk API for batch data import. The following code provides an example:

PUT my_vector_index/_doc/1
{
  "my_text" : "text1",
  "my_vector" : [0.5, 10, 6]
}

PUT my_vector_index/_doc/2
{
  "my_text" : "text2",
  "my_vector" : [-0.5, 10, 10]
}

3. Search for vectors

You can use the Elasticsearch vector similarity search feature to find the documents that are most similar to a query vector. The following code provides an example:

GET my_vector_index/_search
{
  "knn": {
    "field": "my_vector",
    "query_vector": [-5, 9, -12],
    "k": 10,
    "num_candidates": 100
  },
  "fields": [ "my_text" ]
}

Set appropriate parameters

You can set appropriate values for the m and ef_construction parameters. These are advanced parameters for the dense_vector type that you can configure when you create an index. For more information, see Dense vector field type.

Reduce memory consumption

Elasticsearch uses quantization to reduce memory usage. Quantization can reduce the memory size of vectors by a factor of 4, 8, or even 32. For example, a value in a default float vector uses 4 bytes. With int8 quantization, each value uses only 1 byte. With int4 quantization, each value uses half a byte. With Better Binary Quantization (BBQ), each value uses only 1 bit, which means 8 values use 1 byte. Compared to unquantized vectors, BBQ reduces memory requirements to 1/32 of the original size.

To calculate the memory required for vector data:

You need to consider the memory required for both the vector data and the HNSW graph index. The graph index uses a small amount of memory with int8 quantization or no quantization. However, with bbq quantization, the memory used by the graph index increases significantly. When you calculate the memory for vector data, you must account for the memory usage of the graph index.

To calculate vector data memory:

• element_type: float: num_vectors × num_dimensions × 4

• element_type: float with quantization: int8: num_vectors × (num_dimensions + 4)

• element_type: float with quantization: int4: num_vectors × (num_dimensions/2 + 4)

• element_type: float with quantization: bbq: num_vectors × (num_dimensions/8 + 12)

• element_type: byte: num_vectors × num_dimensions

• element_type: bit: num_vectors × (num_dimensions/8)

If you use the flat type and do not create an HNSW index, the vector data memory is calculated as shown in the preceding list. If you use the HNSW type, you must also calculate the size of the graph index. You can estimate the graph index size as follows:

num_vectors × 4 × HNSW.m. The default value of HNSW.m is 16, so the default size is num_vectors × 4 × 16.

The total memory required for vector data is the sum of these two parts.

You must also consider the number_of_replicas setting. The preceding calculation determines the memory size for one copy of the data (one primary shard). The total memory is the size of one copy multiplied by (1 + the number of replicas). The default value for number_of_replicas is 1. This means you have one primary shard and one replica shard, so the total memory required is twice the amount calculated for a single copy.

Check if off-heap memory is sufficient

When you calculate memory capacity and check whether the node memory is sufficient, focus on the off-heap memory of the nodes.

To obtain the amount of off-heap memory, note that a node must reserve sufficient memory for the Java heap. On a node with 64 GB of memory or less, the off-heap memory is typically half of the total memory. On a node with more than 64 GB of memory, the default off-heap memory is the total node memory minus 31 GB. To calculate the exact amount, you can run the following command:

GET _nodes/stats?human&filter_path=**.os.mem.total,**.jvm.mem.heap_max

A node's off-heap memory capacity is calculated as follows: os.mem.total - jvm.mem.heap_max.

Calculate vector index memory

Example:

Assume that you have 10 million 1,024-dimension vectors. You use the default vector settings, enable int8 quantization, set m=16, and use the default number_of_replicas value of 1. The total memory required for the vector data is calculated as follows:

2 × (10,000,000 × (1,024 + 4) + 10,000,000 × 4 × 16) = 20.34 GB.

If you use two data nodes with 16 GB of memory each to store this index, the total available off-heap memory is (16 GB / 2) × 2 = 16 GB. This amount is not sufficient to store the vector data.

If you use two data nodes with 32 GB of memory each, the total available off-heap memory is (32 GB / 2) × 2 = 32 GB. This amount is sufficient to store the vector data.

In a production environment, you must reserve some off-heap memory for other indexes, source documents, and network traffic from read and write operations. Insufficient off-heap memory often causes the disk `ioutil` metric to run at full capacity with high random read traffic.

Prefetch the file system cache

If the machine that runs Elasticsearch restarts, the file system cache is cleared. The operating system then needs time to load the hot areas of the index into memory to accelerate search operations. You can use the index.store.preload setting to explicitly instruct the operating system to immediately load specific files into memory based on their file name extensions.

Example:

PUT /my_vector_index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.store.preload": ["vex", "veq"]
  },
  "mappings": {
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 3
      },
      "my_text" : {
        "type" : "keyword"
      }
    }
  }
}

To determine which index files to load using index.store.preload, see the following descriptions of file name extensions:

The following file name extensions apply to approximate k-NN search. Each extension is subdivided based on the quantization type.

• vex files store the HNSW graph structure.

• vec files contain all non-quantized vector values. This includes all element types, such as floating-point numbers, bytes, and bits.

• veq files are for quantized vectors from indexes that use int4 or int8 quantization.

• veb files are for binary vectors from indexes that use bbq quantization.

• vem, vemf, vemq, and vemb are for metadata. They are usually small and do not need to be preloaded.

Typically, when you use a quantized index, you should preload only the relevant quantized values and the HNSW graph. Preloading the original vectors is not necessary and can be counterproductive.

Configuration examples:

hnsw: "index.store.preload": ["vex", "vec"]

int8, int4: "index.store.preload": ["vex", "veq"]

bbq: "index.store.preload": ["vex", "veb"]

For an existing index, index.store.preload is a static parameter and cannot be changed after the index is created. If you can accept the index being temporarily unavailable, you can set the parameter by closing the index, applying the setting, and then reopening the index. The following example shows how to perform this operation:

POST my_vector_index/_close

PUT my_vector_index/_settings
{
  "index.store.preload": ["vex", "veq"]
}

POST my_vector_index/_open

Reduce the number of index segments

Elasticsearch shards are composed of segments (segment), which are internal storage elements in the index. For an approximate k-NN search, Elasticsearch stores the vector values of each segment as a separate HNSW graph. Therefore, a k-NN search must check each segment. Recent k-NN search parallelization makes searching across multiple segments faster. However, the speed of a k-NN search can still be improved several times over if there are fewer segments. By default, Elasticsearch periodically merges smaller segments into larger ones through a background process. If this is not sufficient, you can take the following explicit steps to reduce the number of index segments.

PUT my_vector_index/_settings
{
  "index.merge.policy.max_merged_segment": "10gb"
}

Exclude vector fields from _source

Elasticsearch stores the original JSON document that was indexed in the _source field. By default, every hit in a search result includes the full _source document. When documents contain high-dimension dense vector fields, the _source can be very large and expensive to load. This can significantly slow down k-NN searches.

You can use the excludes mapping parameter to exclude dense vector fields from being stored in _source. This prevents large amounts of raw vector data from being loaded and returned during a search and reduces the index size. Vectors excluded from _source can still be used in k-NN searches because that process uses an independent data structure. Before you use the excludes parameter, review the potential disadvantages of excluding fields from _source, as described in the preceding note.

PUT /my_vector_index
{
  "mappings": {
    "_source": {
      "excludes": [
        "my_vector"
      ]
    },
    "properties": {
      "my_vector": {
        "type": "dense_vector",
        "dims": 3
      },
      "my_text": {
        "type": "keyword"
      }
    }
  }
}

To view the vector content in a document on Elasticsearch 8.17 or later, you can use the following command:

GET my_vector_index/_search
{
  "docvalue_fields": ["my_vector"]
}

Other versions are also available:

GET my_vector_index/_search
{
  "script_fields": {
    "vector_field": {
      "script": {
        "source" : "doc['my_vector'].vectorValue"
      }
    }
  }
}

As an alternative to excluding vector fields from _source, see synthetic _source.

Upgrade the instance type configuration

Vector similarity calculation is a compute-intensive task that requires high CPU performance. To obtain more than double the performance, select a Turbo instance type. You can use a blue-green deployment to change to a Turbo instance type of the same specification.