:Basic vector search

Prerequisites

You have installed the Proxima CE package and prepared the input table. For more information, see Install the Proxima CE package.

Limitations

Vector search tasks cannot run in a project where the tenant-level schema syntax switch is enabled. If this switch is enabled, an error similar to Schema xxx does not exist is returned. Run the code examples in this topic in a project where this switch is disabled.

Notes

• If you have permissions for an External Volume, you can use the Volume method. Otherwise, provide an AccessKey or a RAM role ARN to allow the Proxima CE task to create and use the required resources.

• When you run a Proxima CE task in DataWorks, you must use the Shared Resource Groups for Scheduling for Smoke Testing. For more information, see Perform smoke testing.

Populate the input table

To test the entire process, run the following commands on a SQL node in DataWorks to generate small data tables for a simple search example.

ALTER TABLE doc_table_float_smoke ADD PARTITION(pt='20221111');
INSERT OVERWRITE TABLE doc_table_float_smoke PARTITION (pt='20221111') VALUES
('1.nid','1~1~1~1~1~1~1~1'),
('2.nid','2~2~2~2~2~2~2~2'),
('3.nid','3~3~3~3~3~3~3~3'),
('4.nid','4~4~4~4~4~4~4~4'),
('5.nid','5~5~5~5~5~5~5~5'),
('6.nid','6~6~6~6~6~6~6~6'),
('7.nid','7~7~7~7~7~7~7~7'),
('8.nid','8~8~8~8~8~8~8~8'),
('9.nid','9~9~9~9~9~9~9~9'),
('10.nid','10~10~10~10~10~10~10~10');
ALTER TABLE query_table_float_smoke ADD PARTITION(pt='20221111');
INSERT OVERWRITE TABLE query_table_float_smoke PARTITION (pt='20221111') VALUES
('q1.nid','1~1~1~1~2~2~2~2'),
('q2.nid','4~4~4~4~3~3~3~3'),
('q3.nid','9~9~9~9~5~5~5~5');

Run the task

You can run the task in DataWorks or odpscmd.

Run in DataWorks

In DataWorks, create a MaxCompute ODPS MR node and run a Proxima CE task using an ODPS SQL script.

• Volume method

  --@resource_reference{"<proxima_ce_jar>"} -- To generate this reference, on the Data Development page, right-click the uploaded JAR package and select Reference Resource.
  jar -resources <proxima_ce_jar>  -- The uploaded Proxima CE JAR package.
  -classpath <proxima_ce_jar> com.alibaba.proxima2.ce.ProximaCERunner  -- The classpath that specifies the entry class of the main function.
  -doc_table doc_table_float_smoke  -- The input doc table.
  -doc_table_partition 20221111  -- The partition of the input doc table.
  -query_table query_table_float_smoke  -- The input query table.
  -query_table_partition 20221111  -- The partition of the input query table.
  -output_table output_table_float_smoke  -- The output table.
  -output_table_partition 20221111  -- The partition of the output table.
  -data_type float  -- The data type of vectors.
  -dimension 8  -- The vector dimension.
  -topk 1  -- The top-k value for vector search.
  -job_mode train:build:seek:recall  -- The mode of the search task. The default is train:build:seek. Add :recall to calculate the recall rate.
  -external_volume_name <ext_volume> -- The External Volume on OSS. The underlying OSS directory must also be created. Otherwise, the task fails.
  -owner_id <oid>  -- A unique ID to identify the user.
  ; -- Do not omit the semicolon. It marks the end of an ODPS SQL statement.

  Replace the following placeholders with your actual values:

  Parameter	Description
  proxima_ce_jar	The name of the uploaded Proxima CE installation package, for example, proxima-ce-aliyun-1.0.1.jar. For more information, see Install the Proxima CE package.
  ext_volume	The name of the External Volume. For more information about creating an External Volume, see External Volume operations.
  oid	A unique numeric user identifier up to 32 digits long, such as 123456. We recommend using your Alibaba Cloud account ID.

• RAM role ARN method

  --@resource_reference{"<proxima_ce_jar>"}  -- To generate this reference, on the Data Development page, right-click the uploaded JAR package and select Reference Resource.
  jar -resources <proxima_ce_jar>  -- The uploaded Proxima CE JAR package.
  -classpath <proxima_ce_jar> com.alibaba.proxima2.ce.ProximaCERunner  -- The classpath that specifies the entry class of the main function.
  -doc_table doc_table_float_smoke  -- The input doc table.
  -doc_table_partition 20221111  -- The partition of the input doc table.
  -query_table query_table_float_smoke  -- The input query table.
  -query_table_partition 20221111  -- The partition of the input query table.
  -output_table output_table_float_smoke  -- The output table.
  -output_table_partition 20221111  -- The partition of the output table.
  -data_type float  -- The data type of vectors.
  -dimension 8  -- The vector dimension.
  -topk 1  -- The top-k value for vector search.
  -job_mode train:build:seek:recall  -- The mode of the search task. The default is train:build:seek. Add :recall to calculate the recall rate.
  -oss_role_arn <rolearn> -- The RAM role ARN with permissions to access OSS. Example: acs:ram::1234xxx5678:role/xxx-role.
  -oss_endpoint <endpoint> -- The endpoint of the destination region.
  -oss_bucket <bucket> -- The OSS bucket.
  -owner_id <oid>  -- A unique ID to identify the user.
  ; -- Do not omit the semicolon. It marks the end of an ODPS SQL statement.

  Replace the following placeholders with your actual values:

  Parameter	Description
  proxima_ce_jar	The name of the uploaded Proxima CE installation package, for example, proxima-ce-aliyun-1.0.1.jar. For more information, see Install the Proxima CE package.
  rolearn	The Alibaba Cloud Resource Name (ARN) of the RAM role, for example, acs:ram::1234xxx5678:role/xxx-role. You can obtain the ARN from the RAM console. In the navigation pane on the left, choose Identities > Role.
  endpoint	The internal endpoint of OSS in the region where the MaxCompute project is located. For more information, see Regions and Endpoints.
  bucket	The name of an OSS bucket in the same region as your MaxCompute project. For information about how to view bucket names, see List buckets.
  oid	A unique numeric user identifier up to 32 digits long, such as 123456. We recommend using your Alibaba Cloud account ID.

Run in odpscmd

Run one of the following scripts in the MaxCompute client (odpscmd):

• Volume method

  jar -resources <proxima_ce_jar> 
  -classpath <proxima_ce_jar_path> com.alibaba.proxima2.ce.ProximaCERunner 
  -doc_table doc_table_float_smoke 
  -doc_table_partition 20221111 
  -query_table query_table_float_smoke 
  -query_table_partition 20221111 
  -output_table output_table_float_smoke 
  -output_table_partition 20221111 
  -data_type float 
  -dimension 8 
  -topk 1 
  -job_mode train:build:seek:recall 
  -external_volume_name <ext_volume> 
  -owner_id <oid> 
  ;

  Replace the following placeholders with your actual values:

  Parameter	Description
  proxima_ce_jar	The name of the uploaded Proxima CE installation package, for example, proxima-ce-aliyun-1.0.1.jar. For more information, see Install the Proxima CE package.
  proxima_ce_jar_path	The local path of the Proxima CE JAR package. If the JAR package is in the same directory as the script, you can use the package name alone.
  ext_volume	The name of the External Volume. For more information about creating an External Volume, see External Volume operations.
  oid	A unique numeric user identifier up to 32 digits long, such as 123456. We recommend using your Alibaba Cloud account ID.

• RAM role ARN method

  jar -resources <proxima_ce_jar> 
  -classpath <proxima_ce_jar_path> com.alibaba.proxima2.ce.ProximaCERunner 
  -doc_table doc_table_float_smoke 
  -doc_table_partition 20221111 
  -query_table query_table_float_smoke 
  -query_table_partition 20221111 
  -output_table output_table_float_smoke 
  -output_table_partition 20221111 
  -data_type float 
  -dimension 8 
  -topk 1 
  -job_mode train:build:seek:recall 
  -oss_role_arn <rolearn> 
  -oss_endpoint <endpoint> 
  -oss_bucket <bucket> 
  -owner_id <oid> 
  ;

  Replace the following placeholders with your actual values:

  Parameter	Description
  proxima_ce_jar	The name of the uploaded Proxima CE installation package, for example, proxima-ce-aliyun-1.0.1.jar. For more information, see Install the Proxima CE package.
  proxima_ce_jar_path	The local path of the Proxima CE JAR package. If the JAR package is in the same directory as the script, you can use the package name alone.
  rolearn	The Alibaba Cloud Resource Name (ARN) of the RAM role, for example, acs:ram::1234xxx5678:role/xxx-role. You can obtain the ARN from the RAM console. In the navigation pane on the left, choose Identities > Role.
  endpoint	The OSS internal endpoint for the region where the MaxCompute project is located. For more information, see Regions and Endpoints.
  bucket	The name of an OSS bucket in the same region as your MaxCompute project. For information about how to view bucket names, see List buckets.
  oid	A unique numeric user identifier up to 32 digits long, such as 123456. We recommend using your Alibaba Cloud account ID.

Results

• Sample console output

  Vector Search  Data Type:4 , Vector Dimension:8 , Search Method:HNSW , Distance Metric:SquaredEuclidean , Build Mode:train:build:seek:recall
  Doc Table Details Table Name: doc_table_float_smoke , Partition:20221111 , Doc Count:10 , Vector Delimiter:~
  Query Table Details Table Name: query_table_float_smoke , Partition:20221111 , Query Count:3 , Vector Delimiter:~
  Output Table Details Table Name: output_table_float_smoke , Partition:20221111
  Row and Column Details  Rows: 1 , Columns:1 , Docs per Column Index:1000000
  Clear Volume Index:false
  Time spent by each worker (seconds):
     SegmentationWorker:      1
     TmpTableWorker:      0
     KmeansGraphWorker:       0
     BuildJobWorker:      120
     SeekJobWorker:       60
     TmpResultJoinWorker:     0
     RecallWorker:        60
     CleanUpWorker:       1
  Total time (minutes):
  Actual recall rate
      Recall@1:   1.0
  

• Sample output table

  +------------+------------+------------+------------+
  | pk         | knn_result | score      | pt         |
  +------------+------------+------------+------------+
  | q1.nid     | 1.nid      | 4.0        | 20221111   |
  | q2.nid     | 3.nid      | 4.0        | 20221111   |
  | q3.nid     | 7.nid      | 32.0       | 20221111   |

Million-scale top-K search

Powered by the Proxima 2.x kernel, Proxima CE delivers high-performance million-scale top-K search, retrieving the top K most similar results for a query vector from a dataset of millions of vectors. To enable this feature, set the -topk startup parameter.