The algorithm experiment feature is available only for Artificial Intelligence Recommendation (AIRec) instances of Algorithm Configuration Edition.
Objectives
To optimize a metric such as the click-through rate (CTR) or the time spent on a page, you must formulate at least two plans and divide the traffic of client users generated within the same period of time into groups. Make sure that each group of client users has the same characteristics. Then, you can see different plan designs. You can determine the optimal experiment plan based on the real data feedback from several groups of client users and then apply the experiment configurations to all online traffic.
Note: Users in this topic refer to client users.
Terms
default experiment
The traffic that is not allocated to an experiment in a scenario belongs to the default experiment. The custom experiment configurations whose recommendation effect meets your expectations can be applied to the default experiment.
apply custom experiment configurations
After an experiment is started and observed for a period of time, if the experiment effect meets your expectations, you can apply the configurations of the current experiment to the default experiment. Then, the default experiment uses the custom experiment configurations.
scenario-based business policy
AIRec provides a set of policies that you can adjust in a scenario based on your business requirements.
operations policy
AIRec provides a set of policies that operations engineers can customize based on business requirements. An operations policy consists of configurations that can adapt to operations requirements and accordingly improve product experience.
algorithm policy
AIRec provides a set of policies that algorithm engineers can customize based on business requirements. An algorithm policy affects the effect of recommendation algorithms. The configurations of an algorithm policy can be applied to the default experiment only after the effect of the configurations is verified by an experiment in a scenario for a period of time.
configuration
An algorithm policy consists of configuration items.
configuration key
A configuration key identifies a configuration item.
filtering table
A table can be created to store the information about items to be filtered, such as the similarity between items for item-based collaborative filtering, item characteristics for the vectors of items, or user characteristics for the vectors of users.
ranking service
AIRec provides an online prediction service based on Elastic Algorithm Service (EAS) of Machine Learning Platform for AI (PAI). You can use this service to score and rank the filtered items online.
trace_info
When you use the experiment management feature, you must make sure that the values of the trace_info field conform to data specifications and are accurately returned in recommendation results. Examples:
(1) If the trace_id field of an item is set to selfhold, set the trace_info field to 1.
(2) If the trace_id field of an item is set to Alibaba, the trace_info field value of this item is returned in the recommendation results.
If the value of the trace_id field for a behavioral data entry is Alibaba, this data entry is generated after this item is recommended by AIRec.
When you upload behavioral data, you can retain the value of the trace_info field for this item.
(3) Sample values of the trace_info field: ST_EDF470CB-D084-4E2C-812E-3F13B9AEA528_4, 1007.5911.12351.1002000:::::
Procedure for creating an experiment
1. You can enable the experiment feature in a scenario only after your instance is started and the scenario is in the running state.
2. Relationships between experiments and scenarios: You can create multiple scenarios for an instance. A maximum of 20 experiments can be created in a scenario. The experiments between scenarios do not interfere with each other.
3. Before you create an experiment, you must enable the experiment feature.
Step 1: Enable the experiment feature
The process of enabling the experiment feature takes about 1 minute.
After you click Enable Experiment Feature, you must select a traffic division method.
If most client users have logged on to your business, we recommend that you divide traffic by user ID.
If most client users are the visitors of your business, we recommend that you divide traffic by device ID (UTDID).
After you click OK, the experiment feature is enabled within about 1 minute. By default, a default experiment is created. If you create an experiment for the first time, the default experiment uses the default recommendation plan of the current scenario.
To view the configurations of the default experiment, click Details in the Actions column. To view the business performance report of the current experiment, click Business Performance Report in the Actions column.
Step 2: Create an experiment
Click Create Experiment.
1. Enter the basic information
2. Configure the experiment
You can modify the filtering and ranking algorithms on the configuration page.
Note: For more information about the introduction and parameters of each algorithm, see Industry algorithm models.
For example, to configure the item-based collaborative filtering algorithm, take note of the following items:
Configure the item-based collaborative filtering algorithm
By default, the algorithm model is enabled. If you need to disable the algorithm model for the experiment, turn off the switch for the algorithm model.
Click Show in the Actions column. Then, you can set the following parameters:
Maximum Number of Retrieval Results for Retrieval Link Based on Collaborative Filtering of Items: the maximum number of items filtered based on the item-based collaborative filtering algorithm.
Optimization Operator and Priority of Retrieval Link Based on Collaborative Filtering of Items: the priorities of various item-based collaborative filtering algorithms.
Customize filtering algorithms
If you purchase an instance of the advanced edition, you can add custom filtering algorithms to algorithm models.
Note: To configure and use custom filtering algorithms, you must create and register a filtering table in advance.
Set the priorities of algorithm models
Customize ranking models
If you purchase an instance of the advanced edition, you can add custom ranking models to algorithm models.
3. Debug the experiment
Before the experiment is started, add test users to the experiment whitelist to verify the effect of your experiment.
Parameters:
(1) Experiment To Be Debugged: the ID of the experiment to be debugged.
(2) Debugging User: the ID of the user that is used to debug the experiment.
(3) Number of Recommended Commodities: the number of items to be recommended. Maximum value: 10.
4. Allocate traffic and start the experiment
Note: We recommend that you allocate 10% to 20% of the total traffic to the current experiment. You can allocate traffic based on your business requirements. Before you start the experiment, you must debug the experiment.
Click Start Experiment.
5. Perform follow-up operations
After the experiment is started, you can click Details, Business Performance Report, and More in the Actions column to view the details of the experiment and perform follow-up operations.
After the experiment is created, you can go to the Experiment Parameter Settings page in the AIRec console to view the experiment list.
Details
Click Details to view the details of the experiment.
Business Performance Report
View the business performance report on the Experiment Effect Analysis page.
Decision-making operations on the experiment
Note: Decision-making operations on an experiment are irreversible. Proceed with caution. The operations include applying custom experiment configurations, stopping the experiment, and deleting the experiment.
Apply custom experiment configurations
We recommend that you observe an experiment for about one week. If the experiment effect meets your expectations, you can apply the configurations of the current experiment to the default experiment. The traffic allocated to the current experiment is also released.
After this operation is performed, the default experiment in the current scenario uses the configurations of the custom experiment. The traffic allocated to the custom experiment now belongs to the default experiment.
For example, the default experiment has configurations A1, B1, and C1, and three traffic buckets numbered from 1 to 3. The business performance reports show that the effect of Experiment X with configurations A2, B2, and C3 is the best. After you apply the configurations of Experiment X to the default experiment, the default experiment has configurations A2, B2, and C3, and six traffic buckets numbered from 1 to 6.
Procedure:
Find the desired experiment and choose More > Make Decision on Experiment in the Actions column.
In the Make Decision on Experiment panel, select Apply Custom Experiment Configurations. The panel displays the configurations of the current experiment that are different from those of the default experiment. Click Synchronize to Default Experiment in the Actions column to apply the custom configurations to the default experiment. Then, click OK.
Stop an experiment
If the effect of an experiment does not meet your expectations, you can select Stop Experiment to end the experiment.