OpenSearch:Stress testing on vector indexes

Preparations

• We recommend performing testing in a Virtual Private Cloud (VPC) environment.

  • For more information about VPCs, see What is a VPC.

  • For more information about how to perform stress testing over the Internet, see Configure the public access whitelist.

• Download the gist-960-euclidean dataset:

  • https://github.com/erikbern/ann-benchmarks

• http://corpus-texmex.irisa.fr/

Decompress the package and use the gist_base.fvecs file.

• Install Python 3 and related libraries.

h5py
json
numpy
sklearn
alibabacloud_ha3engine_vector

Generate data

• Run prepare_data.py. The script supports vector data in .hdf5, .fvecs, .bvecs, and .ivecs format. In this example, we use .hdf5.

python3 prepare_data.py -i ./gist-960-euclidean.hdf5 

• View the data/ subdirectory in the script directory.

  • gist-960-euclidean.hdf5.data is generated.

  • Check the number of data rows.

wc -l data/gist-960-euclidean.hdf5.data

1000000 gist-960-euclidean.hdf5.data

Purchase an OpenSearch Vector Search Edition instance

For more information, see Purchase an OpenSearch Vector Search Edition instance.

Create a table

References:

• Create a table for a MaxCompute data source

• Create a table for an API data source

• Create a table for an OSS data source

• Data Lake Formation (DLF)

Generate queries

• Run the prepare_query.py script to randomly generate queries from the raw data.

python3 prepare_query.py -i gist-960-euclidean.hdf5 -c 10000 -t gist

• Obtain the query.data file that is generated in the data/ subdirectory

Stress testing using wrk

• wrk is an open source HTTP request stress testing tool.

• Download wrk from https://github.com/wg/wrk.

git clone https://github.com/wg/wrk.git

• Run search.lua for stress testing.

  • Copy the script to the wrk/scripts/ directory.

cp search.lua wrk/scripts/

• Calculate the signature and modify header["Authorization"] in the request method.

-- During execution, wrk randomly selects queries to construct specific requests
request = function ()
  local query = query_table[count]
  count = (count + 1)%query_count
  local headers = {}
  headers["Authorization"] = "Basic xxxx" -- Signature information
  headers["Content-Type"] = "application/json"
  return wrk.format("POST", nil, headers, query)
end

• Perform stress testing.

  • -c: the number of concurrent connections.

  • -t: the number of threads for sending the requests.

  • -d: the specified duration for stress testing.

  • -s: the specified script.

  • --latency: displays the detailed stress testing results.

./wrk -c24 -d100s -t8 -s scripts/search.lua http://ha-cn-xxxxxx.ha.aliyuncs.com/vector-service/query --latency

View metrics

• View metrics, such as the recall rate and response time.

  For more information, see Authorize RAM users to view instance monitoring metrics.

Script download links

• prepare_data.py

• push_data.py

• prepare_query.py

• search.lua

Performance testing data

The following shows results of Vector Search Edition, using 8th generation machines, on the ANN_GIST1M 960-dimensional dataset.