Build High-Recall Vector Search with AUTOINDEX - Vector Retrieval Service for Milvus

Vector Retrieval Service for Milvus (Milvus) provides the AUTOINDEX feature. The AUTOINDEX feature in Milvus automates index creation, eliminating the need for manual parameter tuning to achieve optimal search performance. This topic shows you how to apply AUTOINDEX on a collection.

Background information

In Approximate Nearest Neighbor (ANN) search scenarios, AUTOINDEX uses an intelligent algorithm. It analyzes your data distribution during index creation and uses a machine learning model to automatically select retrieval parameters. This process balances the recall rate and retrieval performance. As a result, you do not need to manually set retrieval parameters and can benefit from an optimized search experience.

Compared with open source Milvus, AUTOINDEX in Milvus has significant performance advantages. A benchmark test shows that on a specific dataset, the Queries Per Second (QPS) of AUTOINDEX is three times that of other index types. AUTOINDEX provides excellent performance in the following scenarios:

It uses single instruction, multiple data (SIMD) to accelerate queries and data storage, improving overall server performance.
It optimizes the data graph and pruning strategies to significantly reduce the number of data points accessed during retrieval.
It uses a dynamic quantization policy to effectively reduce the overhead of distance calculations.

Prerequisites

You have installed the PyMilvus library on your local client and updated it to the latest version.
If you have not installed the PyMilvus library or need to update it, run the following command.
```
pip install --upgrade pymilvus
```
You have created a Milvus instance. For more information, see Create a Milvus instance.

Examples

Create a Collection

The following code snippet creates a collection in Milvus without an index by omitting the index parameter. The collection is not loaded into memory upon creation.

from pymilvus import MilvusClient, DataType

client = MilvusClient(
    uri="http://c-xxxx.milvus.aliyuncs.com:19530",
    token="your_user:your_password",
    db_name="default"
)

schema = MilvusClient.create_schema(
    auto_id=False,
    enable_dynamic_field=True,
)

schema.add_field(field_name="id", datatype=DataType.INT64, is_primary=True)
schema.add_field(field_name="vector", datatype=DataType.FLOAT_VECTOR, dim=5)

client.create_collection(
    collection_name="demo", 
    schema=schema, 
)

You can replace the following parameters as needed.

Parameter	Description
`uri`	The endpoint and port of the Milvus instance. The format is `c-xxx.milvus.aliyuncs.com:19530`, where `c-xxx` is the ID of the Milvus instance.
`token`	The username and password to log on to the database. The default username is root.
`db_name`	The database name. The default value is default.
`collection_name`	The name of the collection. This example uses `demo`.

Create an AUTOINDEX

You can configure the index parameter and call the create_index function to create an automatic index (AUTOINDEX).

from pymilvus import MilvusClient

index_params = MilvusClient.prepare_index_params()

# Index parameters for open source Milvus.

# index_params.add_index(
#     field_name="vector",
#     index_type="IVF_FLAT",
#     metric_type="L2",
#     params={"nlist": 1024}
# )

index_params.add_index(
    field_name="vector",
    metric_type="L2",
    index_type="AUTOINDEX",
)

client.create_index(
    collection_name="demo",
    index_params=index_params
)

client.load_collection(
    collection_name="demo"
)

Insert data

You can use the following code to insert test data.

client.insert(
    collection_name="demo",
    data=[
         {"id": 0, "vector": [0.3580376395471989, -0.6023495712049978, 0.18414012509913835, -0.26286205330961354, 0.9029438446296592], "color": "pink_8682"},
         {"id": 1, "vector": [0.19886812562848388, 0.06023560599112088, 0.6976963061752597, 0.2614474506242501, 0.838729485096104], "color": "red_7025"},
         {"id": 2, "vector": [0.43742130801983836, -0.5597502546264526, 0.6457887650909682, 0.7894058910881185, 0.20785793220625592], "color": "orange_6781"},
         {"id": 3, "vector": [0.3172005263489739, 0.9719044792798428, -0.36981146090600725, -0.4860894583077995, 0.95791889146345], "color": "pink_9298"},
         {"id": 4, "vector": [0.4452349528804562, -0.8757026943054742, 0.8220779437047674, 0.46406290649483184, 0.30337481143159106], "color": "red_4794"},
         {"id": 5, "vector": [0.985825131989184, -0.8144651566660419, 0.6299267002202009, 0.1206906911183383, -0.1446277761879955], "color": "yellow_4222"},
         {"id": 6, "vector": [0.8371977790571115, -0.015764369584852833, -0.31062937026679327, -0.562666951622192, -0.8984947637863987], "color": "red_9392"},
         {"id": 7, "vector": [-0.33445148015177995, -0.2567135004164067, 0.8987539745369246, 0.9402995886420709, 0.5378064918413052], "color": "grey_8510"},
         {"id": 8, "vector": [0.39524717779832685, 0.4000257286739164, -0.5890507376891594, -0.8650502298996872, -0.6140360785406336], "color": "white_9381"},
         {"id": 9, "vector": [0.5718280481994695, 0.24070317428066512, -0.3737913482606834, -0.06726932177492717, -0.6980531615588608], "color": "purple_4976"}
     ],
)

Vector search

You can adjust parameters based on the index type. For example, when using an HNSW index, you can adjust the ef parameter. For IVF-based indexes, you can adjust the nprobe parameter. To find the best balance between recall rate and retrieval efficiency, you often need to try different parameter combinations. AUTOINDEX introduces a unified retrieval precision control parameter, level, to simplify parameter tuning. The default value of this parameter is 1, and the maximum value is 5. As the level value increases, the recall rate improves, but retrieval performance may decrease. In most scenarios, the default level value meets the requirements for a recall rate of approximately 90%. If you require a higher recall rate, you can increase the value of the level parameter.

# Retrieval parameters for open source Milvus.
# search_params = {
#     "params": {"nprobe": 10}
# }

search_params = {
    "params": {"level": 1}
}

res = client.search(
    collection_name="demo",
    data=[[0.05, 0.23, 0.07, 0.45, 0.13]],
    limit=3,
    search_params=search_params
)

(Optional) View the Collection

On the Details tab of your Milvus instance, click Attu Manager in the upper-right corner. Then, enter the username and password for the Milvus instance to view the written data, vectors, and other information.