MaxCompute:Inner product and cosine distance

Prerequisites

• The Proxima CE package is installed and a doc table and a query table are created as input tables. For more information, see Install the Proxima CE package.

• Import data to the doc and query tables.

Scenario

You can use MipsSquaredEuclidean or NormalizeConverter to calculate the inner product of vectors.

MipsSquaredEuclidean

If the original vectors cannot be normalized, you can use MipsSquaredEuclidean to convert the dimensions of the original vectors. After the dimensions of the original vectors are converted, the Euclidean distance between the vectors is equivalent to the inner product of the vectors.

If you want to use MipsSquaredEuclidean, you must set the -distance_method parameter to MipsSquaredEuclidean. You can also configure the -measure_params parameter to specify related parameters. Each parameter is a single-line JSON string. The double quotation marks (") of the parameter do not need to be escaped. Spaces are not allowed in the configuration of each parameter. Example:

{"proxima.mips_euclidean.measure.injection_type":0}

For more information about the parameters, see IndexMeasure parameters.

Sample commands

--@resource_reference{"proxima-ce-aliyun-1.0.2.jar"}  -- Reference the JAR package of Proxima CE that you uploaded. Go to the DataStudio page in the DataWorks console. In the left-side navigation pane of the DataStudio page, click Scheduled Workflow. On the left side of the page that appears, choose Business Flow > MaxCompute > Resource. Right-click the JAR package of Proxima CE that you uploaded and select Insert Resource Path from the shortcut menu to display data as an annotation.
jar -resources proxima-ce-aliyun-1.0.2.jar  -- The JAR package of Proxima CE that you uploaded.
-classpath proxima-ce-aliyun-1.0.2.jar com.alibaba.proxima2.ce.ProximaCERunner  -- The classpath specifies the entry class of the main function.
-doc_table doc_table_xx  -- The input doc table.
-doc_table_partition 20221111  -- The partition of the input doc table.
-query_table query_table_xx  -- The input query table.
-query_table_partition 20221111  -- The partition of the input query table.
-output_table output_table_xx  -- The output table.
-output_table_partition 20221111  -- The partition of the output table.
-data_type float  -- The data type of the vector.
-dimension 8 -- The number of dimensions of the vector.
-external_volume_name xxx_volume_name -- The volume on OSS provided by the user must have the underlying OSS directory created in advance; otherwise, the operation will fail.
-owner_id 123456  -- The ID of the user.
-distance_method MipsSquaredEuclidean -- You must configure this parameter.
-measure_params {"proxima.mips_euclidean.measure.injection_type":0};  -- Optional.

NormalizeConverter

If the original vectors can be normalized, you can use NormalizeConverter to perform L2 normalization on the original vectors in the input doc or query table. After L2 normalization is performed, the inner product and the Euclidean distance have the following relationship: . The conversion between the inner product and the Euclidean distance can be performed. In this case, the inner product can be used for distance calculation. After you use NormalizeConverter to perform normalization, you can configure the distance-related parameters to create indexes or run search tasks based on your business requirements.

If you want to use NormalizeConverter, you must set the -converter parameter to NormalizeConverter and set the -distance_method parameter to inner_product. By default, L2 normalization is performed. You can configure other parameters based on your business requirements for vector search. The search result is the inner product after the vector normalization.

Sample commands

--@resource_reference{"proxima-ce-aliyun-1.0.2.jar"}  -- Reference the JAR package of Proxima CE that you uploaded. Go to the DataStudio page in the DataWorks console. In the left-side navigation pane of the DataStudio page, click Scheduled Workflow. On the left side of the page that appears, choose Business Flow > MaxCompute > Resource. Right-click the JAR package of Proxima CE that you uploaded and select Insert Resource Path from the shortcut menu to display data as an annotation.
jar -resources proxima-ce-aliyun-1.0.2.jar  -- The JAR package of Proxima CE that you uploaded.
-classpath proxima-ce-aliyun-1.0.2.jar com.alibaba.proxima2.ce.ProximaCERunner  -- The classpath specifies the entry class of the main function.
-doc_table doc_table_xx  -- The input doc table.
-doc_table_partition 20221111  -- The partition of the input doc table.
-query_table query_table_xx  -- The input query table.
-query_table_partition 20221111  -- The partition of the input query table.
-output_table output_table_xx  -- The output table.
-output_table_partition 20221111  -- The partition of the output table.
-data_type float  -- The data type of the vector.
-dimension 8 -- The number of dimensions of the vector.
-external_volume_name xxx_volume_name -- The volume on OSS provided by the user must have the underlying OSS directory created in advance; otherwise, the operation will fail.
-owner_id 123456  -- The ID of the user.
-converter NormalizeConverter  -- The default normalized converter.
-distance_method inner_product;  -- You must configure this parameter.

Cosine distance