Note: The algorithm experiment feature is available only for Artificial Intelligence Recommendation (AIRec) instances of Standard Edition.
Algorithm configurations for AIRec instances of Standard Edition and new features of the experimental platform
1. Objectives
To optimize a metric such as the click-through rate or dwell time, you need to formulate at least two plans and divide the traffic of client users generated within the same period of time into groups. On the premise that each group of client users has the same characteristics, you can see different plan designs. Then, you can determine the optimal experiment plan based on the real data feedback of several groups of client users and push all the traffic online.
Note: Users in this topic refer to client users.
2. Terms
default experiment
The traffic that is not allocated to an experiment in a scene belongs to the default experiment. The custom experiment configurations whose recommendation effects meet your expectations can be applied to the default experiment.
apply custom experiment configurations
After an experiment is started and observed for a period of time, if the experiment effects meet your expectations, you can apply the configurations of the current experiment to the default experiment. Then, the default experiment uses the custom experiment configurations.
business policy in a scene
AIRec provides a set of policies in a scene that you can adjust based on your business needs.
operation policy
AIRec provides a set of policies that operation engineers can customize based on business needs. An operation policy consists of configurations that can adapt to operational requirements and accordingly improve product experience.
algorithm policy
AIRec provides a set of policies that algorithm engineers can customize based on business needs. An algorithm policy affects the effects of recommendation algorithms. The configurations of an algorithm policy can be applied to the default experiment only after the effects of the configurations are verified by an experiment in a scene for a period of time.
configuration
An algorithm policy consists of configurations.
configuration key
A configuration key identifies a configuration.
recall table
A table can be created to store the information about items to be recalled, such as the similarity between items for item-based collaborative filtering, item characteristics for the vectors of items, or user characteristics for the vectors of users.
sorting service
AIRec provides an online prediction service based on Elastic Algorithm Service (EAS) of Machine Learning Platform for AI (PAI). You can use this service to score and sort recall results online.
trace_info
When you use the experiment management feature, you must make sure that the values of the trace_info field are properly defined in data specifications and properly returned in recommendation results. Examples:
(1) If the trace_id field of an item is set to selfhold, set the trace_info field to 1.
(2) If the trace_id field of an item is set to Alibaba, the trace_info field of this item is returned in recommendation results.
The behavioral data with the trace_id field set to Alibaba indicates that this behavioral data entry is generated after this item is recommended by AIRec.
When you upload behavioral data, you can retain the value of the trace_info field for this item.
(3) Sample values of the trace_info field: ST_EDF470CB-D084-4E2C-812E-3F13B9AEA528_4, 1007.5911.12351.1002000:::::
3. Procedure
You can enable the experiment feature in a scene only after your instance is started and the current scene is in the Running state.
Relationships between experiments and scenes: You can create multiple scenes for an instance. A scene can contain up to 20 experiments. The experiments between scenes do not interfere with each other.
Before you create an experiment, you must enable the experiment feature.
Step 1: Enable the experiment feature
The process of enabling the experiment feature takes about 1 minute.
After you click Enable Experiment Feature, you must select a traffic division method.
If most client users are logged on to your business, we recommend that you divide traffic by user ID.
If most client users are the visitors of your business, we recommend that you divide traffic by UTDID.
After you click OK, the experiment feature is enabled within about 1 minute. By default, a default experiment is created. If you create an experiment for the first time, the default experiment is bound to the default recommendation plan of the current scene.
You can click Details to view the configurations of the default experiment. You can also click Business Performance Report to view the business performance report of the current experiment.
Step 2: Create an experiment
Click Create Experiment.
(1) Enter the basic information
(2) Configure the experiment
You can modify the recall and sorting algorithms on the configuration page.
Note: For more information about the introduction and parameters of each algorithm, see Industry algorithm models.
For example, to configure the item-based collaborative filtering algorithm, take note of the following items:
Configure item-based collaborative filtering
By default, the algorithm model is enabled. If you need to disable the algorithm model for the experiment, turn off the switch.
Click Show in the Actions column. Then, you can set the following parameters:
Maximum Number of Retrieval Results for Retrieval Link Based on Collaborative Filtering of Items: the maximum number of items recalled based on item-based collaborative filtering.
Optimization Operator and Priority of Retrieval Link Based on Collaborative Filtering of Items: the priorities of various item-based collaborative filtering algorithms in descending order.
Set the priorities of algorithm models
(3) Debug the experiment
Before the experiment is started, add test users to the experiment whitelist to verify the effects of your experiment.
Parameters:
Experiment To Be Debugged: the ID of the experiment to be debugged.
Debugging User: the ID of the user that is used to debug the experiment.
Number of Recommended Commodities: the number of commodities to be recommended. Maximum value: 10.
(4) Allocate traffic and start the experiment
Note: We recommend that you allocate 10% to 20% of the total traffic to the current experiment. You can allocate traffic based on your needs. Before you start the experiment, you must debug the experiment.
Click Start Experiment.
(5) Perform follow-up operations
After the experiment is started, you can click Details, Business Performance Report, and More in the Actions column to view the details of the experiment and perform follow-up operations.
After the experiment is created, you can go to the Scene Management page of your instance in the AIRec console, go to the Experiment Management page of your scene, and then view the experiment list.
Details
Click Details to view the details of the experiment.
Business Performance Report
4. Decision-making
Note: Decision-making operations on an experiment are irreversible. For example, apply custom experiment configurations, stop the experiment, or delete the experiment. We recommend that you proceed with caution.
Apply custom experiment configurations
We recommend that you observe an experiment for about one week. If the experiment effects meet your expectations, you can apply the configurations of the current experiment to the default experiment. The traffic allocated to the current experiment is also released.
After this operation, the default experiment in the current scene uses the configurations of the custom experiment. The traffic allocated to the custom experiment now belongs to the default experiment.
For example, the default experiment has configurations A1, B1, and C1, and three traffic buckets numbered from 1 to 3. The business performance reports show that Experiment X with configurations A2, B2, and C3 has the best effects. After you apply the configurations of Experiment X to the default experiment, the default experiment has configurations A2, B2, and C3, and six traffic buckets numbered from 1 to 6.
Procedure:
Find an experiment and choose More > Make Decision on Experiment in the Actions column.
In the Make Decision on Experiment panel, select Apply Custom Experiment Configurations. The panel displays the configurations of the current experiment that are different from those of the default experiment. Click the button in the Actions column to apply the custom configurations to the default experiment. Then, click OK.
Stop an experiment
If the effects of an experiment do not meet your expectations, you can select Stop Experiment to end the experiment.