Artificial Intelligence Recommendation:Retrieval configuration

Hologres

Collaborative filtering requires two tables: a u2i table to retrieve a list of items based on a user_id, and an i2i table to retrieve similar items. The schemas for these two tables have a fixed format.

In addition to retrieving similar items from an i2i table (u2i2i), you can also retrieve them indirectly from i2x and x2i tables (u2i2x2i). In this method, x represents an attribute such as category, brand, or city. The process first retrieves the x attribute of an item, such as the category field. Then, it retrieves popular items that have the corresponding x value from the x2i table to use as recommendations.

u2i table

i2i table (required only for u2i2i)

i2x table (required only for u2i2x2i)

x2i table (required only for u2i2x2i)

Sample u2i2i configuration:

{
    "RecallConfs": [
        {
            "Name": "collaborative_filter",
            "RecallType": "UserCollaborativeFilterRecall",
            "RecallCount": 1000,
            "UserCollaborativeDaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "User2ItemTable": "u2i_table",
                "Item2ItemTable": "i2i_table",
                "Normalization": "on"
            }
        }
    ]
}

Sample u2i2x2i configuration:

{
    "RecallConfs": [
        {
            "Name": "collaborative_filter",
            "RecallType": "UserCollaborativeFilterRecall",
            "RecallCount": 1000,
            "UserCollaborativeDaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "User2ItemTable": "u2i_table",
                "Item2XTable": "i2x_table",
                "X2ItemTable": "x2i_table",
                "XKey": "category",
                "XDelimiter": ",",
                "Normalization": "on"
            }
        }
    ]
}

UserCollaborativeDaoConfig:

PAI-FeatureStore

Collaborative filtering requires two tables: a u2i table to retrieve a list of items based on a user_id, and an i2i table to retrieve similar items. The schemas for these two tables have a fixed format.

The data for both tables is generated in MaxCompute. You must register the data to Feature Store using an offline feature view. The schemas for the MaxCompute tables are as follows:

u2i table

The following figure shows how to register the table to Feature Store:

i2i table

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "collaborative_filter",
            "RecallType": "UserCollaborativeFilterRecall",
            "RecallCount": 1000,
            "UserCollaborativeDaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "User2ItemFeatureViewName": "u2i_recall",
                "Item2ItemFeatureViewName": "i2i_collaborative",
                "Normalization": "on"
            }
        }
    ]
}

UserCollaborativeDaoConfig:

TableStore (OTS)

Collaborative filtering requires two tables: a u2i table to retrieve a list of items based on a user_id, and an i2i table to retrieve similar items. The schemas for these two tables have a fixed format.

u2i table

i2i table

{
    "RecallConfs": [
        {
            "Name": "collaborative_filter",
            "RecallType": "UserCollaborativeFilterRecall",
            "RecallCount": 1000,
            "UserCollaborativeDaoConf": {
                "AdapterType": "tablestore",
                "TableStoreName": "tablestore_info",
                "User2ItemTable": "u2i_table",
                "Item2ItemTable": "i2i_table",
                "Normalization": "on"
            }
        }
    ]
}

Redis

The collaborative_filter process for Redis is unique and involves two steps:

• Construct a key by concatenating RedisPrefix and the UID to query the U2I list. The key returns a string. The supported formats are item_id1,item_id2,item_id3... or item_id1:score1,item_id2:score2,item_id3:score3....

• Query the I2I list. Use the MGET command to query based on the item_ids obtained in the previous step. The I2I data is also a string with the format item_id1:score1,item_id2:score2,item_id3:score3....

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "collaborative_filter",
            "RecallType": "UserCollaborativeFilterRecall",
            "RecallCount": 1000,
            "UserCollaborativeDaoConf": {
                "AdapterType": "redis",
                "RedisName": "redis_info",
                "RedisPrefix": "cr_",
                "Normalization": "on"
            }
        }
    ]
}

Hologres

The data retrieval logic is the same as collaborative filtering, but the U2I data is calculated in real time from a user's historical behavior table.

The user's historical behavior table is updated in real time based on real-time logs, which enables real-time retrieval.

Similar to collaborative filtering, this method also supports u2i2x2i. It first retrieves the x attribute of an item and then retrieves other items with the same x attribute.

User historical behavior table

i2i table (required only for u2i2i)

i2x table (required only for u2i2x2i)

x2i table (required only for u2i2x2i)

Behavior table creation statement:

BEGIN;
CREATE TABLE "sv_rec"."user_behavior_seq" (
  "user_id" text NOT NULL,
  "item_id" text NOT NULL,
  "event" text NOT NULL,
  "play_time" float8  NULL,
  "timestamp" int8 NOT NULL
);
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'orientation', 'column');
call set_table_property('table_name', 'distribution_key', '"user_id"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'clustering_key', '"user_id:asc","timestamp:desc"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'bitmap_columns', '"user_id","event"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'dictionary_encoding_columns', '"user_id:auto","item_id:auto","event"');
CALL SET_TABLE_PROPERTY('"sv_rec"."user_behavior_seq"', 'time_to_live_in_seconds', '2592000');
comment on table "sv_rec"."user_behavior_seq" is 'Real-time user behavior sequence';
comment on column "sv_rec"."user_behavior_seq"."user_id" is 'User ID';
comment on column "sv_rec"."user_behavior_seq"."item_id" is 'Item ID';
comment on column "sv_rec"."user_behavior_seq"."event" is 'Event type';
comment on column "sv_rec"."user_behavior_seq"."play_time" is 'Reading duration, playback duration';
comment on column "sv_rec"."user_behavior_seq"."timestamp" is 'UNIX timestamp, in seconds';
COMMIT;

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "RealTimeEtrecRecall",
            "RecallType": "RealTimeU2IRecall",
            "RecallCount": 200,
            "RealTimeUser2ItemDaoConf": {
                "UserTriggerDaoConf": {
                    "AdapterType": "hologres",
                    "HologresName": "holo_info",
                    "HologresTableName": "user_behavior_table",
                    "WhereClause": "event='xxx'",
                    "Limit": 200,
                    "EventPlayTime": "playback:5000;playvslide:5000",
                    "EventWeight": "playback:1;playvslide:2",
                    "WeightExpression": "exp((-0.2)*((currentTime-eventTime)/3600/24))",
                    "WeightMode": "sum",
                    "NoUsePlayTimeField": false
                },
                "Item2ItemTable": "i2i_table",
                "SimilarItemIdField": "similar_item_ids"
            }
        }
    ]
}

Sample u2i2x2i configuration:

{
    "RecallConfs": [
        {
            "Name": "RealTimeU2I2X2IRecall",
            "RecallType": "RealTimeU2IRecall",
            "RecallCount": 200,
            "RealTimeUser2ItemDaoConf": {
                "UserTriggerDaoConf": {
                    "AdapterType": "hologres",
                    "HologresName": "holo_info",
                    "HologresTableName": "user_behavior_table",
                    "WhereClause": "event='xxx'",
                    "Limit": 200,
                    "EventPlayTime": "playback:5000;playvslide:5000",
                    "EventWeight": "playback:1;playvslide:2",
                    "WeightExpression": "exp((-0.2)*((currentTime-eventTime)/3600/24))",
                    "WeightMode": "sum",
                    "NoUsePlayTimeField": false
                },
                "Item2XTable": "i2x_table",
                "X2ItemTable": "x2i_table",
                "XKey": "category",
                "XDelimiter": ","
            }
        }
    ]
}

RealTimeUser2ItemDaoConf:

PAI-FeatureStore

The data retrieval logic is the same as collaborative filtering, but the U2I data is calculated in real time from a user's historical behavior table.

The user's historical behavior table is updated in real time based on real-time logs, which enables real-time retrieval.

On the PAI-FeatureStore platform, you can use the behavior sequence type in a feature view to store user historical behavior data.

User historical behavior table

i2i table

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "realtimeu2i",
            "RecallType": "RealTimeU2IRecall",
            "RecallCount": 200,
            "RealTimeUser2ItemDaoConf": {
                "UserTriggerDaoConf": {
                    "AdapterType": "featurestore",
                    "FeatureStoreName": "fs_pairec",
                    "FeatureStoreViewName": "user_behavior_table",
                    "TriggerCount": 100,
                    "EventWeight": "click:1;order:1",
                    "WeightExpression": "exp((-0.2)*((currentTime-eventTime)/3600/24))",
                    "WeightMode": "sum",
                    "NoUsePlayTimeField": false,
                    "ItemIdFieldName": "item_id",
                    "EventFieldName": "event",
                    "PlayTimeFieldName": "playtime",
                    "TimestampFieldName": "event_unix_time"
                },
                "Item2ItemFeatureViewName": "i2i_collaborative",
                "SimilarItemIdField": "similar_item_ids"
            }
        }
    ]
}

RealTimeUser2ItemDaoConf:

Hologres

This method finds the corresponding item list based on the user ID. The table definition is based on a convention.

u2i table

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "user2item_recall",
            "RecallType": "UserCustomRecall",
            "RecallCount": 500,
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "user_item_table"
            }
        }
    ]
}

PAI-FeatureStore

This method finds the corresponding item list based on the user ID. The table definition is based on a convention.

The table data is generated in MaxCompute. You must register the data to Feature Store using an offline feature view. The MaxCompute table data schema is as follows:

u2i table

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "user2item_recall",
            "RecallType": "UserCustomRecall",
            "RecallCount": 500,
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "u2i_recall"
            }
        }
    ]
}

TableStore (OTS)

This method finds the corresponding item list based on the user ID. The table definition is based on a convention.

u2i table

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "user2item_recall",
            "RecallType": "UserCustomRecall",
            "RecallCount": 500,
            "DaoConf": {
                "AdapterType": "tablestore",
                "TableStoreName": "ots_info",
                "TableStoreTableName": "user_item_table"
            }
        }
    ]
}

Redis

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "user2item_recall",
            "RecallType": "UserCustomRecall",
            "RecallCount": 500,
            "DaoConf": {
                "AdapterType": "redis",
                "RedisName": "redis_info",
                "RedisPrefix": ""
            }
        }
    ]
}

Hologres

The table format for group hot item retrieval is based on a convention.

group_hot_table

You can assemble the trigger_id based on user features and context information, such as region and device model.

• Concatenate the features in order with underscores (_) to form the trigger_id.

• An empty feature value corresponds to NULL.

• Features that include a Boundaries field require discretization into left-exclusive, right-inclusive intervals. For example, an age range of [20, 30, 40, 50] corresponds to triggers for <=20, 20-30, 30-40, 40-50, and >50.

A user age of 23 corresponds to "20-30".

A null user age corresponds to "NULL".

A user age of 60 corresponds to ">50".

A user age of 19 corresponds to "<=20".

Hologres table example:

This example uses three features: gender, age, and device model.

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "user_group_hot_recall",
            "RecallType": "UserGroupHotRecall",
            "RecallCount": 500,
            "Triggers": [
                {
                    "TriggerKey": "gender"
                },
                {
                    "TriggerKey": "age",
                    "Boundaries": [20,30,40,50]
                },
                {
                    "TriggerKey": "os"
                }
            ],
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "group_hotness_table"
            }
        }
    ]
}

PAI-FeatureStore

The table format for group hot item retrieval is based on a convention.

The table data is generated in MaxCompute. You must register the data to Feature Store using an offline feature view. The MaxCompute table data schema is as follows:

group_hot_table

You can assemble the trigger_id based on user features and context information, such as region and device model.

• Concatenate the features in order with underscores (_) to form the trigger_id.

• An empty feature value corresponds to NULL.

• Features that include a Boundaries field require discretization into left-exclusive, right-inclusive intervals. For example, an age range of [20, 30, 40, 50] corresponds to triggers for <=20, 20-30, 30-40, 40-50, and >50.

A user age of 23 corresponds to "20-30".

A null user age corresponds to "NULL".

A user age of 60 corresponds to ">50".

A user age of 19 corresponds to "<=20".

Table example:

This example uses three features: gender, age, and device model.

This process uses the trigger feature entity.

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "user_group_hot_recall",
            "RecallType": "UserGroupHotRecall",
            "RecallCount": 500,
            "Triggers": [
                {
                    "TriggerKey": "gender"
                },
                {
                    "TriggerKey": "age",
                    "Boundaries": [20,30,40,50]
                },
                {
                    "TriggerKey": "os"
                }
            ],
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "group_hot_recall"
            }
        }
    ]
}

Hologres

The table schema for global hot item retrieval is the same as the schema for group retrieval. The only difference is that the global hot item retrieval table contains only one record, and its trigger_id is -1.

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "UserGlobalHotRecall",
            "RecallType": "UserGlobalHotRecall",
            "RecallCount": 500,
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "global_hotness_table"
            }
        }
    ]
}

PAI-FeatureStore

The table schema for global hot item retrieval is the same as the schema for group retrieval. The only difference is that the global hot item retrieval table contains only one record, and its trigger_id is -1.

The table data is generated in MaxCompute. You must register the data to Feature Store using an offline feature view. The MaxCompute table data schema is as follows:

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "UserGlobalHotRecall",
            "RecallType": "UserGlobalHotRecall",
            "RecallCount": 500,
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "global_hot_recall"
            }
        }
    ]
}

TableStore (OTS)

The table schema for global hot item retrieval is the same as the schema for group retrieval. The only difference is that the global hot item retrieval table contains only one record, and its trigger_id is -1.

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "UserGlobalHotRecall",
            "RecallType": "UserGlobalHotRecall",
            "RecallCount": 500,
            "DaoConf": {
                "AdapterType": "tablestore",
                "TableStoreName": "ots_info",
                "TableStoreTableName": "global_hotness_recall"
            }
        }
    ]
}

Hologres

{
    "RecallConfs": [
        {
            "Name": "AllLiveItemRecall",
            "RecallType": "ColdStartRecall",
            "RecallCount": 3000,
            "ColdStartDaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "item_status_table",
                "WhereClause": "islist_status=1",
                "PrimaryKey": "\"item_id\"",
                "TimeInterval": 0
            }
        }
    ]
}

ColdStartDaoConf:

In the preceding configuration, when you filter the candidate set using WhereClause, you can filter based on the features field passed in the API or the user's feature fields. For example, to filter the candidate set based on language and the user's city attribute, you can write the WhereClause as follows:

"language=${context.features.language} AND city=${user.city}"

In this example, language can be passed in the features field of the API, and city is a user feature.

The source of the parameters is specified in the following format:

• user.xxx represents the value of the xxx feature from the user's features.

• context.features.xxx represents the value of the xxx feature from the features field in the API.

PAI-FeatureStore

Provide a feature view whose primary key is the item ID. A random batch of item IDs is retrieved from this view.

{
    "RecallConfs": [
        {
            "Name": "item_cold_start_recall",
            "RecallType": "ColdStartRecall",
            "RecallCount": 200,
            "ColdStartDaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "rec_sln_demo_item_table_preprocess_all_feature_v1"
            },
            "FilterParams": [
            ]
        }
    ]
}

The parameters are described as follows:

The feature view specified for FeatureStoreViewName can be an offline or a real-time feature view. For a real-time view, this retrieval method obtains new item IDs in real time.

You can use the FilterParams parameter to filter for specific item IDs. If you leave this parameter empty, no filter conditions are set and item IDs are retrieved randomly.

{
            "Name": "item_cold_start_recall",
            "RecallType": "ColdStartRecall",
            "RecallCount": 200,
            "ColdStartDaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "rec_sln_demo_item_table_preprocess_all_feature_v1"
            },
            "FilterParams": [
                {
                    "Name": "click_count",
                    "Type": "int",
                    "Operator": "greater",
                    "Value": "15"
                },
                {
                    "Name": "category",
                    "Type": "string",
                    "Operator": "in",
                    "Value": "user.category"
                }
            ]
        }

In the preceding example, a Value prefixed with user. indicates that the value is retrieved from user features. These features, such as category, can be passed in the features field of the API.

For more information about the Operator, see the Appendix.

Hologres

I2I retrieval is used for similarity-based recommendations. For this type of recommendation, you must pass in the similar item_id. For more information about the specific API definition, see API test.

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "I2IRecall",
            "RecallType": "ItemCollaborativeFilterRecall",
            "RecallCount": 500,
            "ItemCollaborativeDaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "Item2ItemTable": "item_collaborative_list"
            }
        }
    ]
}

Configuration field descriptions:

Definition of the Item2ItemTable table:

PAI-FeatureStore

I2I retrieval is used for similarity-based recommendations. For this type of recommendation, you must pass in the similar item_id. For more information about the specific API definition, see API test.

Definition of the Item2ItemTable table:

Sample configuration:

{
    "RecallConfs": [
        {
            "Name": "I2IRecall",
            "RecallType": "ItemCollaborativeFilterRecall",
            "RecallCount": 200,
            "ItemCollaborativeDaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "i2i_recall"
            }
        }
    ]
}

Configuration field descriptions:

If the item_id passed to the interface is in a multi-value format and is separated by a comma (,), you can use the following configuration:

{
    "RecallConfs": [
        {
            "Name": "I2IRecall",
            "RecallType": "ItemCollaborativeFilterRecall",
            "RecallCount": 200,
            "ItemCollaborativeDaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "i2i_recall"
            },
            "EnableMultipleItemId": true,
            "MultipleItemIdDelimiter": ","
        }
    ]
}