The Vector-based Recall Evaluation component is used to calculate the hit rate of recalls. The hit rate can be used to evaluate the precision of recalls. A higher value indicates a higher precision of recalls that are performed by using the vectors generated during model training. This topic describes how the Vector-based Recall Evaluation component works and how to configure the component.
How it works
The Vector-based Recall Evaluation component supports u2i recalls and i2i recalls. During u2i recalls, user vectors are used to recall the top K items. During i2i recalls, item vectors are used to recall the top K times. The hit rate is calculated by using the following method: For example, the collection of relevant items (user vectors for u2i recalls and item vectors for i2i recalls) that triggers the recall is M. Top K items similar to the trigger items are recalled. N recalled items fall into the M collection. In this scenario, the top K hit rate is calculated based on the |N|/|M| formula. The component also outputs items that do not fall into the M collection and the corresponding distance values for bad case studies. The Vector-based Recall Evaluation component can run in standalone mode or distributed mode. Procedure:
All workers load the user embedding table or item embedding table to build indexes that are required by k-nearest-neighbor (KNN).
The workers look up the true sequence table for the K nearest neighbors in batches based on the embedding table and then return the top K items.
The component calculates the hit rate by comparing the sequence values of the items in the true sequence table with the sequence values of the top K items.
The component aggregates the results and outputs the results to MaxCompute tables.
Item embedding table
An embedding table that stores item vectors. An item embedding table is usually generated by a training algorithm such as GraphSAGE. Example:
item id (bigint)
item embeddings (string)
User embedding table
An embedding table that stores user vectors. A user embedding table is usually generated by a training algorithm such as GraphSAGE. Example:
user id (bigint)
user embeddings (string)
True sequence table
A true table that stores triggers and relevant items. The table is used as the ground truth. For u2i recalls, the trigger id column is mapped to the user id column. For i2i recalls, the trigger id column is mapped to the item id column. In the following true sequence table, the trigger id column is mapped to the item id column.
trigger id (bigint)
item ids (string)
A table that stores the total hit rate values. Example:
A table that stores hit rate details. The hitrate_details table contains the same number of rows as the true sequence table. Example:
The number of rows in this table is the same as that of the true sequence table.
For u2i recalls, the id column is mapped to the user_id column. For i2i recalls, the id column is mapped to the item_id column.
The topk_ids column displays the IDs of the top K items that are relevant to the trigger items. The item IDs are separated by commas (,).
The topk_dists column displays the distance to the items in the topk_ids column.
The hitrate column displays the hit rate of the recalled items corresponding to each trigger.
The bad_ids column displays the items that are recalled but are irrelevant.
The bad_dists column displays the distance to the items in the bad_cases column.
You can configure the Vector-based Recall Evaluation component in the console or by using the CLI. The parameters in the console and at the CLI are the same. The following table describes the parameters.
(Item vector table)
The item embedding table.
(True sequence table)
The true sequence table. For u2i recalls, the table contains users and user-relevant items. For i2i recalls, the table contains items and item-relevant items.
When you evaluate the precision of recalls, if the data in the embedding table is collected at the point in time of T, data in the true sequence table must be collected at the point in time of T+1. Otherwise, the actual hit rate will be higher than the expected value.
(User vector table)
The user embedding table. You need to provide this table only for u2i recalls.
(Vector-based recall hit rate)
An output table that contains the total hit rate values.
(Vector-based recall hit rate details)
An output table that contains the hit rate details.
The type of recall, which can be 'u2i' or 'i2i'.
(Vector table feature dimension)
The embedding dimension of the embedding table.
k (Number of recalled items)
The number of recalled items.
(Optional. Default value: 1)
The method that is used to measure the similarity. If you set the parameter to 0, L2 distance is used to measure the similarity. If you set the parameter to 1, inner products are used to measure the similarity. If L2 distance is used, top K items with the shortest distance are returned. If inner products are used, top K items with the greatest inner products are returned.
strict (Whether to enable the strict mode)
(Optional. Default value: False)
Deviations exist in similarity calculation. If you set the strict parameter to True, the system compares the similarity of vectors in strict mode. However, similarity calculation in this mode is time-consuming.
int (Optional. Default value: 7)
Output The lifecycle of the table in days.
(Optional. Default value: 1024)
The number of samples per batch. Set the parameter to a small value if memory resources are limited.
int (Optional. Default value: 1)
The number of workers that are used to train the model. Set this parameter to a large value when the size of the input table is large or one worker is not enough to train the model efficiently.
(Memory Per core)
(Optional. Default value: 20000)
The amount of memory allocated to each worker. Unit: megabytes. Default: 20000 MB.
pai -name hitrate_gl_ext
The preceding example shows how to calculate the hit rate of u2i recalls. Inner products are used to measure the similarity (distance) of vectors, the strict mode is disabled, and the calculation is performed in batches. One worker is used to process 1,024 items from the true_seq_table table for each batch. The worker is allocated 20 GB of memory and the lifecycle of the hitrate_details and total_hitrate output tables is seven days.