Platform For AI:Create and configure an experiment

Recall experiment example

Step 1: Create a lab

A lab is a collection of traffic. You can create one or more labs. If you create only one lab, it must be the Base lab, which serves as a fallback and is required. Traffic is first matched against non-Base labs. If a recommendation request does not match any non-Base lab, it is routed to the Base lab. You can start by creating a single fallback lab.

On the Experiment Overview page, click the Create Laboratory tab to create a Base lab. Set Bucket Type to UID HASH and Number of Buckets to 100.

Step 2: Set up a layer

Traffic is orthogonal between layers. You can think of a lab as a container and layers as a series of nets within that container. Each time traffic passes through a layer, it is reshuffled before entering the next. This ensures that, with a large enough dataset, the effects of experiments in different layers do not interfere with each other.

On the layer settings page, select UID HASH for Bucketing Method and set Number of Buckets to 100. Set the Traffic Allocation range to 0-99. In Layer Settings, add a new layer. Set Layer Name to recall and Layer Description to Recall layer.

Step 3: Set up an experiment group

Each layer can contain multiple experiment groups, and each group can be assigned a portion of the layer's traffic. This example uses a single experiment group.

On the experiment group configuration page, set the Experiment Group Name, select the Experiment Group Type, set Associated Layer Group to mix_recall_group, and fill in the Description for the experiment group. Below, you can configure the Experiment ID and Experiment Configuration parameters.

Step 4: Set up the experiments

Within an experiment group, you can configure multiple experiments to run concurrently for comparison. Each experiment must have a unique configuration.

Create a baseline experiment

Configure the baseline experiment: Set Experiment Name to hot_group_recall, Description to Group-based hot item recall, and Traffic Allocation to 50%. In the Experiment Configuration, set the recall policy to {"default.RecallNames":["GroupHotRecall"]}.

Create a comparison experiment

Configure the comparison experiment: Set Experiment Name to hot_global_recall, Description to Global hot item recall, and Traffic Allocation to 50%. In the Experiment Configuration, set the recall policy to {"default.RecallNames":["GlobalHotRecall"]}.

Step 5: Debug and observe

In this step, bring the lab, experiment group, and experiments online. Then, send requests to the recommendation engine and check the exp_id field in the response for a value.

If the exp_id value is always empty, verify that the lab, experiment group, and experiments are online. On the Experiment Overview page, an online status is indicated by green, and an offline status is indicated by gray.

A value of ER1_L1#EG1#E1 for exp_id indicates that the request was processed by Experiment 1 and will use its configuration. You can then configure relevant metrics to observe the experiment's performance.

Algorithm configuration experiments

Configuring an algorithm experiment is similar to configuring a recall experiment. To prevent interference between experiments, you can create a new layer, such as rank. The setup for the experiment group and experiments is largely the same; only the experiment configuration needs to be modified.

For example, the baseline experiment configuration can be:

{
    "rankconf": {
        "RankAlgoList": [
            "dbmtl_v1"
        ],
        "RankScore": "${dbmtl_v1_prob}",
        "BatchCount":100,
        "Processor": "EasyRec"
    }
}

The comparison experiment configuration can be:

{
    "rankconf": {
        "RankAlgoList": [
            "dbmtl_v2"
        ],
        "RankScore": "${dbmtl_v2_prob}",
        "BatchCount":100,
        "Processor": "EasyRec"
    }
}