:Configure filters

The PAI-Rec engine provides multiple built-in filter templates, including the templates for the User2ItemExposureFilter, ItemStateFilter, and AdjustCountFilter filters.

Filter configurations

You can configure filters by configuring the FilterConfs parameter in the following sample code. FilterConfs is an array of objects and can be used to define multiple filter policies.

Overview of common filter configurations

The following section describes the common configurations that are referenced by different filters. The configurations are not repeated in the detailed descriptions of each filter in this topic.

Sample configuration:

{
    "FilterConfs": [
        {
            "Name": "",
            "FilterType": "",
            "Dimension": "",
            "DaoConf": {},
            "AdjustCountConfs": [],
            "ItemStateDaoConf": {},
            "FilterParams": [],
            "DiversityDaoConf": {},
            "FilterVal": {}
        }
    ]
}

User2ItemExposureFilter

In many business scenarios, exposure filtering is used to prevent repeated recommendations, typically by combining pseudo-exposure and real exposure.

Pseudo-exposure: The latency of real-time logs makes it impossible to immediately identify which items are exposed. Therefore, the list of items returned by the recommendation engine serves as the pseudo-exposure list.

A real-time computing engine such as Flink writes real-time logs to a database, where they are then consumed by the PAI-Rec engine.

The following common parameters apply to exposure filtering for various data sources.

{
    "FilterConfs": [
        {
            "Name": "holo_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 100,
            "TimeInterval": 172800,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "exposure_history"
            }
        }
    ]
}

BEGIN;
CREATE TABLE "exposure_history" (
 "uid" text NOT NULL,
 "item" text NOT NULL,
 "create_time" int4 NOT NULL
);
CALL SET_TABLE_PROPERTY('"exposure_history"', 'orientation', 'column');
CALL set_table_property('"exposure_history"', 'distribution_key', 'uid');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'clustering_key', '"uid","create_time"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'segment_key', '"create_time"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'bitmap_columns', '"uid","item"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'dictionary_encoding_columns', '"uid","item"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'time_to_live_in_seconds', '172800');
comment on table "exposure_history" is 'Exposure history table';
COMMIT;

{
    "FilterConfs": [
        {
            "Name": "fs_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "TimeInterval": 300,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "user_expose"
            }
        }
    ]
}

"FilterConfs": [
        {
            "Name": "fs_exposure_filter2",
            "FilterType": "User2ItemExposureFilter",
            "WriteLog": false,
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "user_expose"
            }
        }
    ]

{
    "FilterConfs": [
        {
            "Name": "redis_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 100,
            "TimeInterval": 172800,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "redis",
                "RedisName": "redis_info",
                "RedisPrefix": "exposure_"
            }
        }
    ]
}

{
    "FilterConfs": [
        {
            "Name": "ots_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 100,
            "TimeInterval": 172800,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "tablestore",
                "TableStoreName": "tablestore_info",
                "TableStoreTableName": "exposure_history"
            }
        }
    ]
}

Impression de-ranking

When a user's recall pool is small, impression filtering can filter out all available items. To avoid this, tag items that have already been seen rather than filtering them out completely. During reranking, you can then use boosting and demotion to move these tagged items to the end of the list. The following example demonstrates this configuration for a Hologres data source. The process is similar for other data sources.

{
    "FilterConfs": [
        {
            "Name": "holo_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 20,
            "TimeInterval": 172800,
            "WriteLog": true,
            "OnlyLogUserExposeFlag": true,
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "exposure_history"
            }
        }
    ],
    "SortConfs": [
        {
            "Name": "boost_score_sort",
            "SortType": "BoostScoreSort",
            "BoostScoreConditions": [
                {
                    "Conditions": [
                        {
                            "Name": "_is_exposure_",
                            "Domain": "item",
                            "Type": "int",
                            "Value": 1,
                            "Operator": "equal"
                        }
                    ],
                    "Expression": "score / 10"
                }
            ]
        }
    ]
}

This JSON example defines filter and sort configurations. The FilterConfs section uses the holo_exposure_filter to filter out items a user has already been exposed to, based on data from the exposure_history table in Hologres. The SortConfs section uses boost_score_sort for score boosting: if an item has been exposed (_is_exposure_ is 1), its score is divided by 10, reducing its ranking.

User2ItemCustomFilter

{
    "FilterConfs": [
        {
            "Name": "u2i_custom_filter",
            "FilterType": "User2ItemCustomFilter",
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "u2i_custom_filter"
            },
            "ItemStateCacheSize": 10000,
            "ItemStateCacheTime": 3600
        }
    ]
}

{
    "FilterConfs": [
        {
            "Name": "u2i_custom_filter",
            "FilterType": "User2ItemCustomFilter",
            "DaoConf": {
                "AdapterType": "tablestore",
                "TableStoreName": "tablestore_info",
                "TableStoreTableName": "u2i_table"
            }
        }
    ]
}

  {
    "FilterConfs": [
        {
            "Name": "u2i_custom_filter",
            "FilterType": "User2ItemCustomFilter",
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "u2icustom_filter"
            }
        }
    ]
}

AdjustCountFilter

AdjustCountFilter is used to randomly shuffle the items returned by recall links and then retain a specified number of items.

Sample configuration:

{
    "FilterConfs": [
        {
            "Name": "adjust_count_filter",
            "FilterType": "AdjustCountFilter",
            "ShuffleItem": true,
            "RetainNum": 500
        }
    ]
}

PriorityAdjustCountFilter

PriorityAdjustCountFilter is used to control the number of items that are selected from the returned results of recall links based on scores. Each recall link sorts the recommended items based on their scores.

Sample configuration:

{
    "FilterConfs": [
        {
            "Name": "priority_adjust_count_filter",
            "FilterType": "PriorityAdjustCountFilter",
            "AdjustCountConfs": [
                {
                    "RecallName": "recall_1",
                    "Count": 125,
                    "Type": "accumulator"
                },
                {
                    "RecallName": "recall_2",
                    "Count": 250,
                    "Type": "accumulator"
                },
                {
                    "RecallName": "recall_3",
                    "Count": 400,
                    "Type": "accumulator"
                }
            ]
        }
    ]
}

ItemStateFilter

Filtering retrieved items by state requires fetching their states in real time, as they can change at any moment. These states are typically stored in a dedicated table.

{
    "FilterConfs": [
        {
            "Name": "ItemStateFilter",
            "FilterType": "ItemStateFilter",
            "ItemStateDaoConf": {
                "AdapterType": "hologres",
                "HologresName": "",
                "HologresTableName": "",
                "ItemFieldName": "",
                "WhereClause": "",
                "SelectFields": ""
            },
            "ItemStateCacheSize": 50000,
            "ItemStateCacheTime": 3600,
            "FilterParams": [

            ]
        }
    ]
}

{
    "FilterParams": [
        {
            "Name": "publicStatus",
            "Type": "int",
            "Operator": "equal",
            "Value": 0
        },
        {
            "Name": "state",
            "Type": "int",
            "Operator": "equal",
            "Value": 1
        },
        {
            "Name": "checkStatus",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 2
        },
        {
            "Name": "norec",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 1
        }
    ]
}

{
    "FilterConfs": [
        {
            "Name": "ItemStateFilter",
            "FilterType": "ItemStateFilter",
            "ItemStateDaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "",
                "FeatureStoreViewName": "",
                "ItemFieldName": "",
                "SelectFields": ""
            },
            "ItemStateCacheSize": 50000,
            "ItemStateCacheTime": 3600,
            "FilterParams": [

            ]
        }
    ]
}

{
    "FilterParams": [
        {
            "Name": "publicStatus",
            "Type": "int",
            "Operator": "equal",
            "Value": 0
        },
        {
            "Name": "state",
            "Type": "int",
            "Operator": "equal",
            "Value": 1
        },
        {
            "Name": "checkStatus",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 2
        },
        {
            "Name": "norec",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 1
        }
    ]
}

SnakeFilter

The recall stage retrieves a candidate set from multiple recall channels. SnakeFilter combines data from these channels in round-robin order based on specified weight ratios.

For example, if there are three recall channels (A, B, and C) with weights of 1, 2, and 3, respectively, the combination logic works as follows:

• Select one element from candidate set A.

• Select two elements from candidate set B.

• Select three elements from candidate set C.

• Repeat this process until you have the required number of elements.

There are two policies for handling duplicate elements:

• REFILL_ON_DUPLICATE: If an element retrieved from a candidate set duplicates a previously selected one, the system searches the same candidate set for a unique replacement. For example, if the three elements retrieved from candidate set C duplicate elements already selected from sets A and B, the system searches set C for replacements.

• SKIP_ON_DUPLICATE: The system skips any duplicate elements and moves on to the next iteration. For example, if the three elements retrieved from candidate set C duplicate elements from sets A and B, candidate set C returns no elements in this round. The process then starts a new iteration, beginning with candidate set A.

{
    "FilterConfs": [
        {
            "Name": "SnakeFilter",
            "FilterType": "SnakeFilter",
            "RetainNum": 20,
            "SnakeType": "REFILL_ON_DUPLICATE",
            "AdjustCountConfs": [
                 {
                     "RecallName": "GroupHotRecall",
                     "Weight": 1
                 },
                 {
                     "RecallName": "U2IRecall",
                     "Weight": 2
                 },
                 {
                     "RecallName": "GlobalHotRecall",
                     "Weight": 3
                 }
             ]
        }
    ]
}

CompletelyFairFilter

CompletelyFairFilter is used to sort items returned by recall links based on the scores of the items, and then select items from the results of each link in a fair manner.

{
    "FilterConfs": [
        {
            "Name": "CompletelyFairFilter",
            "FilterType": "CompletelyFairFilter",
            "RetainNum": 500
        }
    ]
}

DimensionFieldUniqueFilter

DimensionFieldUniqueFilter works in a different way from UniqueFilter. DimensionFieldUniqueFilter removes items with duplicate field values.

{
    "FilterConfs": [
        {
            "Name": "DimensionFieldUniqueFilter",
            "FilterType": "DimensionFieldUniqueFilter",
            "Dimension": ""
        }
    ]
}

ConditionFilter

This is a composite filter that dynamically executes a specific filter based on request conditions, such as user features or those passed in the context. You can configure one or more combinations, each linking a set of conditions to a specific filter defined in FilterConfs.

The preceding example defines two standalone filters: adjust_count_filter and CompletelyFairFilter. ConditionFilter is a composite filter that routes requests based on conditions. When the user attribute query equals "1", it uses adjust_count_filter. If no condition matches, it falls back to the filter specified by DefaultFilterName (CompletelyFairFilter).

The configuration parameters are as follows:

FilterConfs configuration:

Configure FilterParamConfig as follows:

For details on the condition settings, see the appendix.

UniqueFilter

UniqueFilter is designed to ensure that each item ID is unique. If the same item ID is returned by two recall links, UniqueFilter prioritizes the item ID that is first returned.

You can use UniqueFilter when you configure the FilterNames parameter without the need to configure UniqueFilter.

Filter usage

Similar to the recall configuration, the filter configuration uses the FilterNames parameter. This parameter is a Map[string]object that maps each scenario to a set of filter policies.

• ${scene_name}: The name of the scenario. To apply the same configuration to multiple scenarios, use default.

• UniqueFilter: A custom name for a filter defined in FilterConfigs.

Appendix

Operator examples

1. equal (equal to the specified value)

{
    "Name": "publicStatus",
    "Type": "int",
    "Operator": "equal",
    "Value": 0
}

2. not_equal (not equal to the specified value)

{
    "name": "checkStatus",
    "type": "int",
    "operator": "not_equal",
    "value": 2
}

3. greater (greater than the specified value)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "greater",
    "Value": 2
}

4. greaterThan (greater than or equal to the specified value)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "greaterThan",
    "Value": 2
}

5. less (less than the specified value)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "less",
    "Value": 2
}

6. lessThan (less than or equal to the specified value)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "lessThan",
    "Value": 2
}

7. in (matches any value in an array)

String

{
    "Name": "state",
    "Type": "string",
    "Operator": "in",
    "Value": ["success","ok"]
}

8. not_in (does not match any item in the array)

{
    "name": "state",
    "type": "int",
    "operator": "not_in",
    "value": [2,4,6]
}

You can also use the string type and configure it with the in operator.

9. contains (matches any item in an array)

{
    "Name": "state",
    "Type": "[]int",
    "Operator": "contains",
    "Value": [2,4,6]
}

string type

{
    "Name": "state",
    "Type": "[]string",
    "Operator": "contains",
    "Value": ["success","ok"]
}

10. not_contains (does not contain any of the specified items)

{
    "Name": "state",
    "Type": "[]int",
    "Operator": "not_contains",
    "Value": [2,4,6]
}

The following code uses the not_contains operator to filter for items where the state field does not contain any value from the array [2, 4, 6]. This filter is useful for excluding specific items from a query result. For example, you could use this filter to retrieve all tasks except those with a status of 'archived' (ID 2), 'on-hold' (ID 4), or 'deprecated' (ID 6).

You can also use a string array. For configuration details, see the contains operator.

11. bool (Combines multiple condition-matching operators)

Combine conditions with or.

{
    "Operator": "bool",
    "Type": "or",
    "Configs":
    [
        {
            "Name": "publicStatus",
            "Type": "int",
            "Operator": "equal",
            "Value": 0
        },
        {
            "Name": "checkStatus",
            "Type": "int",
            "Operator": "greater",
            "Value": 2
        }
    ]
}

Using and is similar: set Type to and.

12. expression (Matches when the expression evaluates to true)

{
    "Operator": "expression",
    "Value": "item.size == 43"
}

item.size is an attribute of item; to access an attribute of user, use user.xxx.

Expressions support arithmetic, comparison, logical, and ternary operators. The following is a more complex example:

{
    "Operator": "expression",
    "Value": "!item.sold_out and user.list != nil ? item.size in user.list : true"
}

For a complete definition of the expression syntax, refer to https://expr-lang.org/docs/language-definition.