Artificial Intelligence Recommendation:Configure filters

The PAI-Rec engine provides multiple built-in filter templates, including the templates for the User2ItemExposureFilter, ItemStateFilter, and AdjustCountFilter filters.

Filter configurations

You can configure filters by configuring the FilterConfs parameter in the following sample code. FilterConfs is an array of objects and can be used to define multiple filter policies.

Overview of common filter configurations

The following section describes the common configurations that are referenced by different filters. The configurations are not repeated in the detailed descriptions of each filter in this topic.

Sample configuration:

    "FilterConfs":[
        {
            "Name":"",
            "FilterType":"",
            "Dimension":"",
            "DaoConf":{},
            "AdjustCountConfs":[{}],
            "ItemStateDaoConf":{},
            "FilterParams":[{}],
            "DiversityDaoConf":{},
            "FilterVal":{}
        }
    ]

User2ItemExposureFilter

Exposure blocking is required in many business scenarios to prevent items from being repeatedly recommended. In this case, exposure refers to pseudo-exposure.

Due to the latency of real-time logs, the exposed items cannot be identified immediately. Therefore, the items returned by the recommendation engine are considered as pseudo-exposed items.

The PAI-Rec engine is used to read and write pseudo-exposure data. Real exposure data is obtained from the real-time logs of users and written to a database by using a real-time computing engine such as Apache Flink. Then, PAI-Rec can consume the data.

The following table describes the common parameter configurations of different data sources for User2ItemExposureFilter.

Hologres

"FilterConfs" :[
  {
    "Name": "holo_exposure_filter",
    "FilterType": "User2ItemExposureFilter",
    "MaxItems": 100,
    "TimeInterval": 172800,
    "WriteLog": true,
    "DaoConf":{
       "AdapterType": "hologres",
       "HologresName": "holo_info",
       "HologresTableName": "exposure_history"
    }
  }
]

Parameters of DaoConf

You can set the time_to_live_in_seconds parameter based on your business requirements to define the exposure table.

BEGIN;
CREATE TABLE "exposure_history" (
 "uid" text NOT NULL,
 "item" text NOT NULL,
 "create_time" int4 NOT NULL,
PRIMARY KEY ("uid","create_time")
);
CALL SET_TABLE_PROPERTY('"exposure_history"', 'orientation', 'column');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'clustering_key', '"uid","create_time"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'segment_key', '"create_time"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'bitmap_columns', '"uid","item"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'dictionary_encoding_columns', '"uid","item"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'time_to_live_in_seconds', '172800');
comment on table "exposure_history" is 'the table that stores exposure records';
COMMIT;

ApsaraDB for Redis

"FilterConfs" :[
  {
    "Name": "redis_exposure_filter",
    "FilterType": "User2ItemExposureFilter",
    "MaxItems": 100,
    "TimeInterval": 172800,
    "WriteLog": true,
    "DaoConf":{
       "AdapterType": "redis",
       "RedisName": "redis_info",
       "RedisPrefix": "exposure_"
    }
  }
]

Parameters of DaoConf

Tablestore

"FilterConfs" :[
  {
    "Name": "ots_exposure_filter",
    "FilterType": "User2ItemExposureFilter",
    "MaxItems": 100,
    "TimeInterval": 172800,
    "WriteLog": true,
    "DaoConf":{
       "AdapterType": "tablestore",
       "TableStoreName": "tablestore_info",
       "TableStoreTableName": "exposure_history"
    }
  }
]

Parameters of DaoConf

The exposure table is defined by using the following parameters:

time_to_live_in_seconds: the lifecycle of the data. You must specify a custom value for the parameter.

User2ItemCustomFilter

You need to provide a custom user-to-item (U2I) table for filtering data.

Tablestore

"FilterConfs" :[
  {
    "Name": "u2i_custom_filter",
    "FilterType": "User2ItemCustomFilter",
    "DaoConf":{
       "AdapterType": "tablestore",
       "TableStoreName": "tablestore_info",
       "TableStoreTableName": "u2i_table"
    }
  }
]

Parameters of DaoConf

The exposure table is defined by using the following parameters.

AdjustCountFilter

AdjustCountFilter is used to randomly shuffle the items returned by recall links and then retain a specified number of items.

Sample configuration:

"FilterConfs" :[
{
      "Name": "adjust_count_filter",
      "FilterType": "AdjustCountFilter",
      "ShuffleItem": true,
      "RetainNum": 500
}
]

PriorityAdjustCountFilter

PriorityAdjustCountFilter is used to control the number of items that are selected from the returned results of recall links based on scores. Each recall link sorts the recommended items based on their scores.

Sample configuration:

"FilterConfs" :[
  {
        "Name": "priority_adjust_count_filter",
        "FilterType": "PriorityAdjustCountFilter",
        "AdjustCountConfs" :[
        {
            "RecallName" :"recall_1",
            "Count" :125,
            "Type" : "accumulator"
        },
        {
            "RecallName" :"recall_2",
            "Count" :250,
            "Type" : "accumulator"
        },
        {
            "RecallName" :"recall_3",
            "Count" :400,
            "Type" : "accumulator"
        }
    ]
  }
]

ItemStateFilter

ItemStateFilter is used to filter items returned by recall links based on their states. The state of an item may change in real time. In most cases, a specific table is used to store item states. In this case, the item states need to be queried in real time before filtering.

Hologres

"FilterConfs" :[
  {
    "Name": "ItemStateFilter",
    "FilterType": "ItemStateFilter",
    "ItemStateDaoConf":{
        "AdapterType": "hologres",
        "HologresName": "",
        "HologresTableName": "",
        "ItemFieldName" : "",
        "WhereClause": "",
        "SelectFields" :""
     },
     "FilterParams" :[]
  }
]

Parameters of ItemStateDaoConfig

Parameters of FilterParams

        "FilterParams" :[
            {
                "Name" : "publicStatus",
                "Type" : "int",
                "Operator" : "equal",
                "Value" : 0
            },
            {
                "Name" : "state",
                "Type" : "int",
                "Operator" : "equal",
                "Value" : 1
            },
            {
                "Name" : "checkStatus",
                "Type" : "int",
                "Operator" : "not_equal",
                "Value" : 2
            },
            {
                "Name" : "norec",
                "Type" : "int",
                "Operator" : "not_equal",
                "Value" : 1
            }
        ]

For more information about how to use operators, see the Appendix section of this topic.

CompletelyFairFilter

CompletelyFairFilter is used to sort items returned by recall links based on the scores of the items, and then select items from the results of each link in a fair manner.

"FilterConfs" :[
  {
    "Name": "CompletelyFairFilter",
    "FilterType": "CompletelyFairFilter",
  	"RetainNum": 500
  }
]

DimensionFieldUniqueFilter

DimensionFieldUniqueFilter works in a different way from UniqueFilter. DimensionFieldUniqueFilter removes items with duplicate field values.

"FilterConfs" :[
  {
    "Name": "GroupWeightCountFilter",
    "FilterType": "GroupWeightCountFilter",
  	"Dimension:""
  }
]

UniqueFilter

UniqueFilter is designed to ensure that each item ID is unique. If the same item ID is returned by two recall links, UniqueFilter prioritizes the item ID that is first returned.

You can use UniqueFilter when you configure the FilterNames parameter without the need to configure UniqueFilter.

Filter usage

The usage of filters is similar to that of recall links. After you complete filter configurations, you can configure the FilterNames parameter for a specific scenario. FilterNames is in the Map<String, Object> structure. The key indicates a scenario, and the value indicates a set of filter policies.

"FilterNames": {
  "default": [
    "UniqueFilter"
  ]
}

• default: the name of the scenario. If the scenario is not explicitly configured, default is used.

• UniqueFilter: the custom name of the filter specified in the FilterConfis parameter.

Appendix

Operator examples

1. equal (equal to the specified value)

{
 "Name" : "publicStatus",
 "Type" : "int",
 "Operator" : "equal",
 "Value" : 0
}

2. not_equal (not equal to the specified value)

{
 "Name" : "checkStatus",
 "Type" : "int",
 "Operator" : "not_equal",
 "Value" : 2
}

3. greater (greater than the specified value)

{
 "Name" : "checkStatus",
 "Type" : "int",
 "Operator" : "greater",
 "Value" : 2
}

4. greaterThan (greater than or equal to the specified value)

{
 "Name" : "checkStatus",
 "Type" : "int",
 "Operator" : "greaterThan",
 "Value" : 2
}

5. less (less than the specified value)

{
 "Name" : "checkStatus",
 "Type" : "int",
 "Operator" : "less",
 "Value" : 2
}

6. lessThan (less than or equal to the specified value)

{
 "Name" : "checkStatus",
 "Type" : "int",
 "Operator" : "lessThan",
 "Value" : 2
}

7. in (equal to one value or some values in the array)

{
 "Name" : "state",
 "Type" : "int",
 "Operator" : "in",
 "Value" : [2, 4, 6]
}

String

{
 "Name" : "state",
 "Type" : "string",
 "Operator" : "in",
 "Value" : ["success", "ok"]
}