：過濾配置

PAI-Rec 引擎已經內建了多個過濾模板，包括曝光過濾、狀態過濾、數量調整過濾等。

如何配置

過濾的配置對應配置總覽中的 FilterConfs，FilterConfs 是一個[]object 結構，可以定義多個過濾策略。

過濾公用配置一覽

每種召回配置，都會用到公用配置中的一部分，在此統一解釋，單獨的召回配置中則不再贅述。

配置樣本：

{
    "FilterConfs": [
        {
            "Name": "",
            "FilterType": "",
            "Dimension": "",
            "DaoConf": {},
            "AdjustCountConfs": [],
            "ItemStateDaoConf": {},
            "FilterParams": [],
            "DiversityDaoConf": {},
            "FilterVal": {}
        }
    ]
}

曝光過濾（User2ItemExposureFilter）

很多業務情境都需要曝光過濾來避免重複推薦，通常的曝光過濾做法是偽曝光和真實曝光的組合。

偽曝光：因為即時日誌有延時等原因，我們並不能立刻知道哪些 item 被曝光，所以我們把Recommendation Engine返回的 item 列表作為偽曝光列表。

真實曝光：需要客戶的即時日誌，通過Flink 這種Realtime Compute引擎寫入資料庫，然後被PAI-Rec 引擎消費。

不同資料來源的曝光過濾有以下幾個公用參數配置：

{
    "FilterConfs": [
        {
            "Name": "holo_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 100,
            "TimeInterval": 172800,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "exposure_history"
            }
        }
    ]
}

BEGIN;
CREATE TABLE "exposure_history" (
 "uid" text NOT NULL,
 "item" text NOT NULL,
 "create_time" int4 NOT NULL
);
CALL SET_TABLE_PROPERTY('"exposure_history"', 'orientation', 'column');
CALL set_table_property('"exposure_history"', 'distribution_key', 'uid');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'clustering_key', '"uid","create_time"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'segment_key', '"create_time"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'bitmap_columns', '"uid","item"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'dictionary_encoding_columns', '"uid","item"');
CALL SET_TABLE_PROPERTY('"exposure_history"', 'time_to_live_in_seconds', '172800');
comment on table "exposure_history" is '曝光記錄表';
COMMIT;

{
    "FilterConfs": [
        {
            "Name": "fs_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "TimeInterval": 300,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "user_expose"
            }
        }
    ]
}

"FilterConfs": [
        {
            "Name": "fs_exposure_filter2",
            "FilterType": "User2ItemExposureFilter",
            "WriteLog": false,
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "user_expose"
            }
        }
    ]

{
    "FilterConfs": [
        {
            "Name": "redis_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 100,
            "TimeInterval": 172800,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "redis",
                "RedisName": "redis_info",
                "RedisPrefix": "exposure_"
            }
        }
    ]
}

{
    "FilterConfs": [
        {
            "Name": "ots_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 100,
            "TimeInterval": 172800,
            "WriteLog": true,
            "DaoConf": {
                "AdapterType": "tablestore",
                "TableStoreName": "tablestore_info",
                "TableStoreTableName": "exposure_history"
            }
        }
    ]
}

曝光物品沉底

在某些情境下，使用者的召回池很小，如果配置了曝光過濾，很容易把召回池中的item全部過濾掉，這裡我們可以不具體的過濾，只是對曝光過的物品打一個標籤，然後通過提降權把曝光過的物品排在最後，這裡我們以Hologres資料來源舉例，其他資料來源類似，具體的過濾和重排配置樣本如下：

{
    "FilterConfs": [
        {
            "Name": "holo_exposure_filter",
            "FilterType": "User2ItemExposureFilter",
            "MaxItems": 20,
            "TimeInterval": 172800,
            "WriteLog": true,
            "OnlyLogUserExposeFlag": true,
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "exposure_history"
            }
        }
    ],
    "SortConfs": [
        {
            "Name": "boost_score_sort",
            "SortType": "BoostScoreSort",
            "BoostScoreConditions": [
                {
                    "Conditions": [
                        {
                            "Name": "_is_exposure_",
                            "Domain": "item",
                            "Type": "int",
                            "Value": 1,
                            "Operator": "equal"
                        }
                    ],
                    "Expression": "score / 10"
                }
            ]
        }
    ]
}

User2Item自訂過濾（User2ItemCustomFilter）

{
    "FilterConfs": [
        {
            "Name": "u2i_custom_filter",
            "FilterType": "User2ItemCustomFilter",
            "DaoConf": {
                "AdapterType": "hologres",
                "HologresName": "holo_info",
                "HologresTableName": "u2i_custom_filter"
            },
            "ItemStateCacheSize": 10000,
            "ItemStateCacheTime": 3600
        }
    ]
}

{
    "FilterConfs": [
        {
            "Name": "u2i_custom_filter",
            "FilterType": "User2ItemCustomFilter",
            "DaoConf": {
                "AdapterType": "tablestore",
                "TableStoreName": "tablestore_info",
                "TableStoreTableName": "u2i_table"
            }
        }
    ]
}

  {
    "FilterConfs": [
        {
            "Name": "u2i_custom_filter",
            "FilterType": "User2ItemCustomFilter",
            "DaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "fs_pairec",
                "FeatureStoreViewName": "u2icustom_filter"
            }
        }
    ]
}

數量調整過濾（AdjustCountFilter）

簡單的調整數量，把召回數量隨機打散，然後保留需要控制的數量。

配置樣本：

{
    "FilterConfs": [
        {
            "Name": "adjust_count_filter",
            "FilterType": "AdjustCountFilter",
            "ShuffleItem": true,
            "RetainNum": 500
        }
    ]
}

優先順序數量調整過濾（PriorityAdjustCountFilter）

優先順序過濾可以對各路召回進行數量控制，每路召回都會根據召回的 score 進行排序

配置樣本：

{
    "FilterConfs": [
        {
            "Name": "priority_adjust_count_filter",
            "FilterType": "PriorityAdjustCountFilter",
            "AdjustCountConfs": [
                {
                    "RecallName": "recall_1",
                    "Count": 125,
                    "Type": "accumulator"
                },
                {
                    "RecallName": "recall_2",
                    "Count": 250,
                    "Type": "accumulator"
                },
                {
                    "RecallName": "recall_3",
                    "Count": 400,
                    "Type": "accumulator"
                }
            ]
        }
    ]
}

狀態過濾（ItemStateFilter）

我們經常需要對召回的資料，進行狀態過濾。item 的狀態有可能會即時變動的，一般會有專有的表格儲存體 item 狀態。 此過濾流程需要即時擷取 item 狀態資訊，然後進行過濾。

{
    "FilterConfs": [
        {
            "Name": "ItemStateFilter",
            "FilterType": "ItemStateFilter",
            "ItemStateDaoConf": {
                "AdapterType": "hologres",
                "HologresName": "",
                "HologresTableName": "",
                "ItemFieldName": "",
                "WhereClause": "",
                "SelectFields": ""
            },
            "ItemStateCacheSize": 50000,
            "ItemStateCacheTime": 3600,
            "FilterParams": [

            ]
        }
    ]
}

{
    "FilterParams": [
        {
            "Name": "publicStatus",
            "Type": "int",
            "Operator": "equal",
            "Value": 0
        },
        {
            "Name": "state",
            "Type": "int",
            "Operator": "equal",
            "Value": 1
        },
        {
            "Name": "checkStatus",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 2
        },
        {
            "Name": "norec",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 1
        }
    ]
}

{
    "FilterConfs": [
        {
            "Name": "ItemStateFilter",
            "FilterType": "ItemStateFilter",
            "ItemStateDaoConf": {
                "AdapterType": "featurestore",
                "FeatureStoreName": "",
                "FeatureStoreViewName": "",
                "ItemFieldName": "",
                "SelectFields": ""
            },
            "ItemStateCacheSize": 50000,
            "ItemStateCacheTime": 3600,
            "FilterParams": [

            ]
        }
    ]
}

{
    "FilterParams": [
        {
            "Name": "publicStatus",
            "Type": "int",
            "Operator": "equal",
            "Value": 0
        },
        {
            "Name": "state",
            "Type": "int",
            "Operator": "equal",
            "Value": 1
        },
        {
            "Name": "checkStatus",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 2
        },
        {
            "Name": "norec",
            "Type": "int",
            "Operator": "not_equal",
            "Value": 1
        }
    ]
}

SnakeFilter

在召回階段可以通過多種召回通道擷取候選集。SnakeFilter提供一種根據權重比例，輪流的從各個召回通道拼接資料的方式。

比如召回有3個通道 A，B, C ，配置的權重比例為 1， 2， 3。那麼組裝邏輯為：

• 先從A候選集中取1個元素

• 從B候選集中擷取2個元素

• 從C候選集中擷取3個元素

• 迴圈擷取，直至找到滿足數量的元素

當遇到重複元素時，目前通過兩種策略來控制：

• REFILL_ON_DUPLICATE 如果當前候選集裡擷取的元素和之前取的重複，會在當前候選集中繼續尋找，直至找到不重複的元素。比如C候選集中擷取的3個元素都和A,B裡重複了，會繼續在C候選集裡尋找。

• SKIP_ON_DUPLICATE 直接跳過重複的元素，不補足元素，繼續下一輪迭代。比如C候選集中擷取的3個元素都和A,B裡重複了，那麼本輪C候選集返回的元素為空白，重新一輪迭代，繼續從A尋找。

{
    "FilterConfs": [
        {
            "Name": "SnakeFilter",
            "FilterType": "SnakeFilter",
            "RetainNum": 20,
            "SnakeType": "REFILL_ON_DUPLICATE",
            "AdjustCountConfs": [
                 {
                     "RecallName": "GroupHotRecall",
                     "Weight": 1
                 },
                 {
                     "RecallName": "U2IRecall",
                     "Weight": 2
                 },
                 {
                     "RecallName": "GlobalHotRecall",
                     "Weight": 3
                 }
             ]
        }
    ]
}

CompletelyFairFilter

根據召回條目的 score 進行排序，然後公平的從每一路召回擷取資料

{
    "FilterConfs": [
        {
            "Name": "CompletelyFairFilter",
            "FilterType": "CompletelyFairFilter",
            "RetainNum": 500
        }
    ]
}

DimensionFieldUniqueFilter

和 UniqueFilter 不同， 根據 item 的某個欄位進行去重過濾

{
    "FilterConfs": [
        {
            "Name": "DimensionFieldUniqueFilter",
            "FilterType": "DimensionFieldUniqueFilter",
            "Dimension": ""
        }
    ]
}

ConditionFilter

這個是組合 filter，可以根據請求條件（user 特徵， context 傳入的 features 等）來決定走哪個 filter。可以設定一組或者多組 condition 條件 + 具體 filter （在 FilterConfs 裡已經定義好了）的組合。

{
"FilterConfs":[
  {
      "Name": "adjust_count_filter",
      "FilterType": "AdjustCountFilter",
      "ShuffleItem": true,
      "RetainNum": 500
  },
  {
    "Name": "CompletelyFairFilter",
    "FilterType": "CompletelyFairFilter",
    "RetainNum": 500
  },
  {
    "Name": "ConditionFilter",
    "FilterType": "ConditionFilter",
    "ConditionFilterConfs":{
      "FilterConfs":[
        {
          "Conditions":[
            {
		"Name":     "query",
		"Type":     "string",
		"Domain":   "user",
		"Operator": "equal",
		"Value":    "1"
            }
          ],
          "FilterName":"adjust_count_filter"
        }
      ],
      "DefaultFilterName": "CompletelyFairFilter"
    }
  }
]
}

上文定義了兩個獨立的 filter：adjust_count_filter 和 CompletelyFairFilter。ConditionFilter 是組合條件的filter，當 user 屬性 query == "1" 時，走 adjust_count_filter 的邏輯。如果條件沒有匹配到，走 DefaultFilterName 指定的filter，在本情境中會走 CompletelyFairFilter。

具體配置含義如下：

FilterConfs 具體配置含義：

FilterParamConfig 配置如下：

具體的條件設定，請參考下面的附錄。

UniqueFilter

唯一性過濾，保證 item id 是唯一的，如果兩路召回包含同一個 item id，哪路召回先返回，取哪一路的 item id。

無需配置，可直接在 FilterNames 中引用。

如何使用

過濾配置和召回配置類似，配置好後，提供一個分情境使用的 FilterNames，FilterNames 是一個 Map[string]object 結構，其中 key 是情境，每個情境對應一組過濾策略

"FilterNames": {
  "${scene_name}": [
    "UniqueFilter"
  ]
}

• ${scene_name} 為情境名，如果多個情境想使用同一個配置，則可以使用 "default"。

• UniqueFilter: 此參數為在 FilterConfigs 中定義的過濾自訂名稱。

附錄

條件匹配 Operator 樣本

1. equal (相等 ==）

{
    "Name": "publicStatus",
    "Type": "int",
    "Operator": "equal",
    "Value": 0
}

2. not_equal （不等於 !=)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "not_equal",
    "Value": 2
}

3. greater （大於 >)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "greater",
    "Value": 2
}

4. greaterThan (大於等於 >=)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "greaterThan",
    "Value": 2
}

5. less （小於 <)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "less",
    "Value": 2
}

6. lessThan (小於等於 <=)

{
    "Name": "checkStatus",
    "Type": "int",
    "Operator": "lessThan",
    "Value": 2
}

7. in （與數組中某個條目匹配）

{
    "Name": "state",
    "Type": "int",
    "Operator": "in",
    "Value": [2,4,6]
}

string 類型

{
    "Name": "state",
    "Type": "string",
    "Operator": "in",
    "Value": ["success","ok"]
}

8. not_in （不與數組中任何條目匹配）

{
    "Name": "state",
    "Type": "int",
    "Operator": "not_in",
    "Value": [2,4,6]
}

也可以使用 string 類型，配置可以參考 in Operator。

9. contains （包含數組中任意一個條目）

{
    "Name": "state",
    "Type": "[]int",
    "Operator": "contains",
    "Value": [2,4,6]
}

string 類型

{
    "Name": "state",
    "Type": "[]string",
    "Operator": "contains",
    "Value": ["success","ok"]
}

10. not_contains （不包含數組中任意一個條目）

{
    "Name": "state",
    "Type": "[]int",
    "Operator": "not_contains",
    "Value": [2,4,6]
}

也可以使用 string 數群組類型，配置可以參考 contains Operator。

11. bool （串連多個條件匹配 Operator）

使用 或 （or）的方式串連

{
    "Operator": "bool",
    "Type": "or",
    "Configs":
    [
        {
            "Name": "publicStatus",
            "Type": "int",
            "Operator": "equal",
            "Value": 0
        },
        {
            "Name": "checkStatus",
            "Type": "int",
            "Operator": "greater",
            "Value": 2
        }
    ]
}

使用 且 （and）的方式串連同理，只需要把 Type 設定為 and。

12. expression （運算式結果為 true 時匹配）

{
    "Operator": "expression",
    "Value": "item.size == 43"
}

item.size 表示來自 item 的屬性，如果需要用到 user 屬性，可以用 user.xxx 表示。

運算式中支援使用各種算術運算子、比較子、邏輯運算子，三目運算子等，一個更加複雜的例子如下：

{
    "Operator": "expression",
    "Value": "!item.sold_out and user.list != nil ? item.size in user.list : true"
}

完整的運算式文法支援可以參考： https://expr-lang.org/docs/language-definition。