A/B testing service integration and examples - Artificial Intelligence Recommendation

Integrate the A/B testing feature

Client in the preceding figure refers to services provided by customers. Customers can configure experiments in the PAI-Rec console.
After customers integrate the PAI-Rec engine, the system will automatically pull configurations of experiments after the PAI-Rec engine is started. When recommendation requests are sent to the PAI-Rec engine, SDKs match experiments and obtain specific experiment parameters. With different experiment parameters, the PAI-Rec engine implements different recalls or calls different ranking models.

The PAI-Rec engine predefines some parameters that can be used when you configure an experiment. You must configure experiment parameters based on the rules for parameter names. Otherwise, the parameters cannot be identified after an experiment is matched.

Parameter	Type	Description	Example
default.RecallNames	json array	The names of all recalls.	"default.RecallNames":[ "HomepageEtrecRecall", "HomepageDssmRecall"]
"recall." + the name of the specific recall	json object	The recall that is created based on the recall configurations.	{"recall.MyRecall":{"version":"v2"}}
filterNames	json array	The names of all filters.	{"filterNames":["UniqueFilter", "UserExposureFilter"]}
default.SortNames	json array	The names of all re-rankings.	{"default.SortNames":["ItemRankScore"]}
rankconf	recconf.RankConfig	The configurations of the ranking algorithm.	"rankconf":{"RankAlgoList":["pai_homepage_fm"],"RankScore":"${pai_homepage_fm}"}
features.scene.name	string	The name of the scenario where feature loading is configured.	"homepagetf"
generalRankConf	recconf.GeneralRankConfig	The configurations of the coarse ranking, including the configurations for retrieving user features and the configurations of the RankConf parameter.	{"generalRankConf":{"FeatureLoadConfs":[{"FeatureDaoConf":{}}],"RankConf":{},"ActionConfs":[]}}
coldStartGeneralRankConf	recconf.ColdStartGeneralRankConfig	The configurations of the coarse ranking for cold start.	{"coldStartGeneralRankConf":{"FeatureLoadConfs":[{"FeatureDaoConf":{}}],"RankConf":{},"ActionConfs":[]}}
coldStartRankConf	recconf.ColdStartRankConfig	The configurations of the ranking algorithm specified for the recall in cold start.	{"coldStartRankConf":{"RecallName":"ColdStartRecall", "AlgoName":"linucb"}}

Construct a recommendation context for a request and match the context with an experiment. You can use the following sample code:

        func (c *HomeFeedController) makeRecommendContext() {
        c.context = context.NewRecommendContext()
        c.context.Size = c.param.Limit
        c.context.Param = &c.param
        c.context.RecommendId = c.RequestId
        c.context.Config = recconf.Config
        c.context.Debug = c.param.Debug
        abcontext := model.ExperimentContext{
                Uid:         c.param.DistinctId,
                RequestId:   c.RequestId,
                FilterParams: map[string]interface{}{},
        }

        if abtest.GetExperimentClient() != nil {
                c.context.ExperimentResult = abtest.GetExperimentClient().MatchExperiment(c.param.SceneId, &abcontext)
                log.Info(c.context.ExperimentResult.Info())
        }
}

Adjust the experiment parameters in the sample code. If the recommendation context exists, you can obtain experiment parameters by using context.ExperimentResult.

        count := r.recallCount
        if context.ExperimentResult != nil {
                count = context.ExperimentResult.GetLayerParams("").GetInt("recall.base.count", count)
        }
        fmt.Println("test count", count)

GetLayerParams is used to retrieve all experiment parameters of a specific layer. Get, GetInt, GetFloat, and GetInt64 are used to retrieve specific parameters. In get() functions, the first argument is the parameter name, and the second argument is the default value. If functions fail to find actual parameter values, default values are returned.

Retrieve parameters

The PAI-Rec engine allows you to specify custom parameters to change the engine behaviors when the engine runs. This way, the engine status can be changed without restarting the engine.

// Retrieve the name of the scenario.
scene := context.GetParameter("scene").(string)

// Retrieve the parameters based on the scenario, and then use a Get* function to obtain the specific parameter values.
count := abtest.GetParams(scene).GetInt("count", 100)
fmt.Println("recall count:", count)