Perform performance tests for vector analysis - AnalyticDB

This topic describes performance tests for vector analysis in AnalyticDB for PostgreSQL.

Test environment

To prevent errors caused by network fluctuations, make sure that an AnalyticDB for PostgreSQL instance and an Elastic Compute Service (ECS) instance are deployed in the same virtual private cloud (VPC).

AnalyticDB for PostgreSQL specifications

Engine version	Node specifications of High-performance Edition	Number of compute nodes	Storage capacity of compute nodes	Storage type of compute nodes
V6.6.2.5	8 cores, 32 GB	2	1,000 GB	PL1 Enterprise SSD (ESSD)

ECS specifications

CPU	Memory	Disk storage
16 cores	32 GB	2 TB

Preparations

Prepare a test environment

Install Python 3.8 or later on your on-premises device.
Download the ann-benchmarks test tool of AnalyticDB for PostgreSQL to your on-premises device. Click adbpg_ann_benchmark_20250218.tar. to obtain the installation package.
Run the following command to install the dependencies of the ann-benchmarks test tool:
```
pip install -r requirements.txt 
```
Install Docker of a version later than 20. For more information, see Install Docker Desktop on Linux.
Run the following command to build a test image:
```
python install.py --proc 4 --algorithm adbpg
```

Prepare a test dataset

Download a dataset and store the dataset in the data directory of the ann-benchmarks project.

Dataset	Number of dimensions	Number of samples	Metric function	Dataset parameter	Download URL
GIST	960	1,000,000	L2 similarity	gist-960-euclidean	GIST
SIFT-10M	128	10,000,000	L2 similarity	sift-128-euclidean	SIFT-10M
SIFT-100M	128	100,000,000	L2 similarity	sift100m-128-euclidean	SIFT-100M
Deep	96	10,000,000	Cosine similarity	deep-image-96-angular	Deep
Cohere	768	1,000,000	L2 similarity	cohere-768-euclidean	Cohere
Dbpedia	1536	1,000,000	Cosine similarity	dbpedia-openai-1000k-angular	Dbpedia

Test procedure

Step 1: Configure the connection information of the test tool

Enter configuration information in the ann_benchmarks/algorithms/adbpg/module.py file of the test tool.

# The internal endpoint of the AnalyticDB for PostgreSQL instance. 
self._host = 'gp-bp10ofhzg2z****-master.gpdb.rds.aliyuncs.com'

# The port number of the AnalyticDB for PostgreSQL instance. 
self._port = 5432

# The name of the database in the AnalyticDB for PostgreSQL instance. 
self._dbname = '<database_name>'

# The name of the database account of the AnalyticDB for PostgreSQL instance. 
self._user = '<user_name>'

# The password of the database account of the AnalyticDB for PostgreSQL instance. 
self._password = '<YOUR_PASSWORD>'

Step 2: Configure test parameters

Use the test dataset to edit the ann_benchmarks/algorithms/adbpg/config.yml file of the test tool.

float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 120}]
        query_args: [[ {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 1}, 
        {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 5}, 
        {ef_search: 400, max_scan_points: 3200, pq_amp: 10, parallel: 10}, 
        {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 15}, 
        {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 20}, 
        {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 25}, 
        {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 30}, 
        {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 50}]]

arg_groups: the parameters related to index creation. For information about how to create a vector index, see Create a vector index.

Parameter	Description
M	The M parameter that is used to create a Hierarchical Navigable Small World (HNSW) index. Greater values indicate longer construction time and higher construction precision.
efConstruction	The parameter that is used to determine the search quality.
parallel_build	The parallelism that is used to create the index. In most cases, set this parameter to the number of CPU cores of compute nodes.
external_storage	The policy that is used to cache indexes. Valid values: 1: mmap. 0: shared_buffer. Important Only AnalyticDB for PostgreSQL V6.0 supports the external_storage parameter.
pq_enable	Specifies whether to enable the product quantization (PQ) feature. Valid values: 1: yes. 0: no.
pq_segments	The number of compute nodes that are used for PQ. In most cases, set this parameter to the `number of dimensions divided by 8`.

query_args: the search-related parameters.

Parameter	Description
ef_search	The number of the nearest neighbors that are used for HNSW index search.
max_scan_points	The maximum number of samples examined in a scan.
pq_amp	The PQ amplification factor. If you do not enable the PQ feature, this parameter does not take effect.
parallel	The number of concurrent queries. This parameter takes effect only in batch mode.

During the test, you can modify the preceding parameters to ensure a 95% recall rate. AnalyticDB for PostgreSQL provides the following reference configurations for the preceding test datasets. You can select the parameter configuration for a specific test dataset.

# for gist 960
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 120}]
        query_args: [[ {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 1}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 3200, pq_amp: 10, parallel: 10}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 15}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 20}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 25}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 30}, {ef_search: 100, max_scan_points: 3200, pq_amp: 10, parallel: 50}]]

# for deep 96
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 12}]
        query_args: [[ {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 1}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 10}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 15}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 20}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 25}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 30}, {ef_search: 400, max_scan_points: 1500, pq_amp: 10, parallel: 50}]]

# for cohere 768
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 96}]
        query_args: [[ {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 1}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 10}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 15}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 20}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 25}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 30}, {ef_search: 400, max_scan_points: 600, pq_amp: 10, parallel: 50}]]


# for dbpedia 1536
float:
  any:
  - base_args: ['@metric']
    constructor: ADBPG
    disabled: false
    docker_tag: ann-benchmarks-adbpg
    module: ann_benchmarks.algorithms.adbpg
    name: adbpg
    run_groups:
      nopq_mmap:
        arg_groups: [{M: 64, efConstruction: 600, parallel_build: 8, external_storage: 1, pq_enable: 0, pq_segments: 192}]
        query_args: [[ {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 1}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 5}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 10}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 15}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 20}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 25}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 30}, {ef_search: 400, max_scan_points: 425, pq_amp: 10, parallel: 50}]]

Step 3: Test the search recall rate

After you configure the preceding parameters, run the following command to test the recall rate:

nohup python run.py --algorithm adbpg --dataset <Dataset> --runs 1 --timeout 990000 
> annbenchmark_deep.log 2>&1 &

Note

Replace the value of the dataset parameter with the actual test dataset.

After the test is complete, run the following command to query the recall rate test result:

python plot.py --dataset <Dataset>  --recompute

Sample output:

0:    ADBPG(m=64, ef_construction=600, ef_search=400, max_scan_point=500, pq_amp=10)        recall: 0.963       qps: 126.200
1:   ADBPG(m=64, ef_construction=600, ef_search=400, max_scan_point=1000, pq_amp=10)        recall: 0.992       qps: 122.665

Check whether the recall rate meets your business requirements. If not, modify the parameters and re-execute the test.

Step 4: Test the search performance

After you obtain a recall rate that meets your business requirements, you can perform performance tests in the same manner that you perform the recall rate test. You must enable the batch mode to check the concurrency performance.

nohup python run.py --algorithm adbpg --dataset <Dataset> --runs 1 --timeout 990000 --
batch > annbenchmark_deep.log 2>&1 &

After the test is complete, open the annbenchmark_deep.log file to check the queries per second (QPS), average response time (RT), and P99 RT parameters based on different concurrency conditions.

2023-12-20 17:31:39,297 - INFO - query using 25 parallel
worker 0 cost 9.50 s, qps 315.92, mean rt 0.00317, p99 rt 0.00951
2023-12-20 17:31:49,097 - INFO - QPS: 7653.155
2023-12-20 17:31:49,113 - INFO - query using 30 parallel
worker 0 cost 13.87 s, qps 216.36, mean rt 0.00462, p99 rt 0.04298
2023-12-20 17:32:03,260 - INFO - QPS: 6361.819
2023-12-20 17:32:03,281 - INFO - query using 50 parallel
worker 0 cost 20.78 s, qps 144.36, mean rt 0.00693, p99 rt 0.02735
2023-12-20 17:32:24,385 - INFO - QPS: 7107.920

Test results

This section describes the performance test results of different datasets based on different AnalyticDB for PostgreSQL vector database configurations and index creation modes. In the performance tests, the recall rate is set to greater than or equal to 95% and the top 10 search results are obtained. The following table describes the index creation modes.

Index creation mode	Description	Applicable scenario
PQ + mmap	Uses the mmap policy to cache and persistently store vector indexes and the PQ method to compress vector codes and accelerate vector computing.	1. More than 1,000,000 vectors are involved. 2. A small number of update and delete operations are performed. 3. Memory resources are insufficient to cache all vectors. 4. A moderate performance is required.
noPQ + mmap	Uses the mmap policy to cache and persistently store vector indexes and does not use the PQ method to compress vector codes and accelerate vector computing.	1. A small number of update and delete operations are performed. 2. Memory resources are sufficient to cache all vectors and indexes. 3. The best performance is required.
PQ + shared_buffer	Uses the shared_buffer policy of PostgreSQL to cache vector indexes and the PQ method to compress vector codes and accelerate vector computing.	1. More than 1,000,000 vectors are involved. 2. A large number of update and delete operations are performed. 3. Memory resources are insufficient to store all vectors. 4. A moderate performance is required.
noPQ + shared_buffer	Uses the shared_buffer policy of PostgreSQL to cache vector indexes, and does not use the PQ method to compress vector codes and accelerate vector computing.	1. Less than 1,000,000 vectors are involved. 2. Memory resources are sufficient to store all vectors and indexes. 3. A large number of update and delete operations are performed. 4. A moderate performance is required.

Instance specifications: 8 cores and 32 GB memory × 2 compute nodes

Dataset: GIST L2 (960 × 1,000,000)

Index creation mode: noPQ + mmap

Index creation parameters:
- M: 64
- efConstruction: 600
- parallel_build: 8
- external_storage: 1
- pq_enable: 0
Search parameters:
- ef_search: 100
- max_scan_points: 3200
The following table describes the test results.
Index creation time (s)
Query concurrency
QPS
Average RT (ms)
P99 RT (ms)
485
1
396
1
2
5
1,744
2
3
10
3,073
2
4
15
3,358
3
10
20
3,511
5
15
25
3,601
6
21
30
3,689
7
25
50
3,823
12
36

Dataset: Deep IP (96 × 10,000,000)

Index creation mode: noPQ + mmap

Index creation parameters:
- M: 64
- efConstruction: 600
- parallel_build: 8
- external_storage: 1
- pq_enable: 0
Search parameters:
- ef_search: 400
- max_scan_points: 1500
The following table describes the test results.
Index creation time (s)
Query concurrency
QPS
Average RT (ms)
P99 RT (ms)
1,778
1
878
1
2
5
4,344
1
2
10
7,950
1
3
15
10,114
1
4
20
10,629
1
5
25
10,858
2
7
30
11,093
2
9
50
11,354
4
16

Dataset: Cohere L2 (768 × 1,000,000)

Index creation mode: noPQ + mmap

Index creation parameters:
- M: 64
- efConstruction: 600
- parallel_build: 8
- external_storage: 1
- pq_enable: 0
Search parameters:
- ef_search: 400
- max_scan_points: 600
The following table describes the test results.
Index creation time (s)
Query concurrency
QPS
Average RT (ms)
P99 RT (ms)
465
1
561
1
2
5
2,893
1
2
10
5,108
1
3
15
5,488
2
5
20
5,969
2
8
25
6,195
3
12
30
6,098
4
19
50
6,138
7
39

Dataset: Dbpedia IP (1,536 × 1,000,000)

Index creation mode: noPQ + mmap

Index creation parameters:
- M: 64
- efConstruction: 600
- parallel_build: 8
- external_storage: 1
- pq_enable: 0
Search parameters:
- ef_search: 400
- max_scan_points: 425
The following table describes the test results.
Index creation time (s)
Query concurrency
QPS
Average RT (ms)
P99 RT (ms)
807
1
453
1
2
5
1,948
1
3
10
2,820
2
4
15
2,903
4
11
20
2,860
6
19
25
2,897
7
27
30
2,880
9
34
50
2,877
16
63

Index creation time (s)	Query concurrency	QPS	Average RT (ms)	P99 RT (ms)
485	1	396	1	2
	5	1,744	2	3
	10	3,073	2	4
	15	3,358	3	10
	20	3,511	5	15
	25	3,601	6	21
	30	3,689	7	25
	50	3,823	12	36