Platform For AI:How to integrate the A/B test system with your system

Log on to the PAI-Rec console

In the PAI-Rec console, click Full-link Service in the left-side navigation pane. In the Service Initialization wizard that appears, configure your workspaces.

Configuration notes:

1. Query existing workspaces from Platform for AI (PAI), DataWorks, and MaxCompute in their respective consoles. On each product homepage, click Console.

2. Create an OSS bucket in the OSS console before configuring it here.

Grant MaxCompute access

On the Projects page, search for the MaxCompute project you want to use.

If no project appears, select the correct region first.

In the Actions column of the project, click Manage. Grant PAI-Rec direct access and write permissions on the MaxCompute project.

For details, see Service activation and initialization.

Create a lab

1. Select a runtime environment and recommendation scenario.

2. Click Create Lab.

3. On the Create Lab panel, configure the following parameters:

Create an experiment group

Each experiment layer can have multiple experiment groups, and each group can have multiple experiments. Use separate groups when multiple engineers need to run experiments on the same layer (for example, different recall algorithms) without interfering with each other.

A/A test group (recommended for new experiments):

To validate that your experiment setup is unbiased, create an A/A test group before running real experiments:

1. Select 4–5 buckets for the experiment.

2. Assign 2 buckets to a group with no policy configured. Run these buckets as a dry run and monitor key metrics.

3. Use the two groups whose metrics are most similar for your actual A/B test.

Additional experiment group parameters:

Create an experiment

Click Create Experiment. By default, a baseline experiment is created first.

Traffic allocation strategies:

After configuring all parameters, click Save.

Python SDK

pip install https://aliyun-pairec-config-sdk.oss-cn-hangzhou.aliyuncs.com/python/aliyun_pairec_config_python_sdk-1.0.0-py2.py3-none-any.whl

export ALIBABA_CLOUD_ACCESS_KEY_ID=<your-access-key-id>
export ALIBABA_CLOUD_ACCESS_KEY_SECRET=<your-access-key-secret>

import os
from alibabacloud_tea_openapi.models import Config
from api.api_scene import SceneApiService
from api.api_experiment_room import ExperimentRoomApiService
from api.api_layer import api_layer
from api.api_experiment_group import ExperimentGroupApiService
from api.api_experiment import ExperimentApiService
from client.client import ExperimentClient
from model.experiment import ExperimentContext
from api.api_crowd import CrowdApiService
from alibabacloud_pairecservice20221213.client import Client
from common.constants import ENVIRONMENT_PRODUCT_CONFIG_CENTER
from common.constants import ENVIRONMENT_PREPUB_CONFIG_CENTER
from common.constants import ENVIRONMENT_DAILY_CONFIG_CENTER

# Instance details — find these on the Basic Information page in the PAI-Rec console
instance_id = "<instance-id>"
region = "<region-id>"

# Read credentials from environment variables
access_id = os.environ['ALIBABA_CLOUD_ACCESS_KEY_ID']
access_key = os.environ['ALIBABA_CLOUD_ACCESS_KEY_SECRET']

if __name__ == '__main__':
    # Valid environments: ENVIRONMENT_PRODUCT_CONFIG_CENTER, ENVIRONMENT_PREPUB_CONFIG_CENTER, ENVIRONMENT_DAILY_CONFIG_CENTER
    experiment = ExperimentClient(
        instance_id=instance_id,
        region=region,
        access_key_id=access_id,
        access_key_secret=access_key,
        environment=ENVIRONMENT_PRODUCT_CONFIG_CENTER
    )

    # Build the request context
    # request_id: any unique identifier (auto-increment ID or UUID)
    # uid: the user ID to assign to an experiment
    experiment_context = ExperimentContext(
        request_id="<request-id>",
        uid="<user-id>",
        filter_params={}
    )

    # Match the user to an experiment in the specified scenario
    # Find the scenario name on the Recommendation Scenarios page
    experiment_result = experiment.match_experiment("<scenario-name>", experiment_context)

    # Log experiment assignment details
    print('info', experiment_result.info())

    # Store exp_id in your event tracking logs for metric calculation
    print('exp_id', experiment_result.get_exp_id())

    # Read experiment parameters to determine which algorithm variant to run
    print(experiment_result.get_experiment_params())
    print(experiment_result.get_experiment_params().get('url', 'not exist'))
    print(experiment_result.get_experiment_params().get('token', 'not exist'))

info exp_id=<exp_id> ...
exp_id <exp_id>
{'url': 'https://...', 'token': '...'}
https://...
...

Java SDK

package com.aliyun.openservices.pairec;

import com.aliyun.openservices.pairec.api.ApiClient;
import com.aliyun.openservices.pairec.api.Configuration;
import com.aliyun.openservices.pairec.common.Constants;
import com.aliyun.openservices.pairec.model.ExperimentContext;
import com.aliyun.openservices.pairec.model.ExperimentResult;

public class ExperimentTest {
    static ExperimentClient experimentClient;

    public static void main(String[] args) throws Exception {
        // Instance details — find regionId and instanceId on the Basic Information page
        String regionId = "<region-id>";
        String instanceId = System.getenv("Instance ID"); // pai-rec instance ID

        // Read credentials from environment variables
        String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
        String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");

        Configuration configuration = new Configuration(regionId, accessId, accessKey, instanceId);
        // Set the experiment environment
        configuration.setEnvironment(Constants.Environment_Product_Desc);

        ApiClient apiClient = new ApiClient(configuration);
        experimentClient = new ExperimentClient(apiClient);
        // Initialize the client
        experimentClient.init();

        // Build the request context
        ExperimentContext experimentContext = new ExperimentContext();
        experimentContext.setUid("<user-id>");
        experimentContext.setRequestId("<request-id>");

        // Match the user to an experiment in the specified scenario
        ExperimentResult experimentResult = experimentClient.matchExperiment("<scenario-name>", experimentContext);

        // Store exp_id in your event tracking logs for metric calculation
        System.out.println(experimentResult.getExpId());
        // Log experiment assignment details
        System.out.println(experimentResult.info());

        // Read experiment parameters to determine which algorithm variant to run
        System.out.println(experimentResult.getExperimentParams().getString("rank_version", "not exist"));
        System.out.println(experimentResult.getExperimentParams().getString("version", "not exist"));
        System.out.println(experimentResult.getExperimentParams().getString("recall", "not exist"));
        System.out.println(experimentResult.getExperimentParams().getDouble("recall_d", 0.0));

        // Read parameters for a specific experiment layer by name
        System.out.println(experimentResult.getLayerParams("recall").getString("rank_version", "not exist"));
        System.out.println(experimentResult.getLayerParams("rank").getString("version", "not exist"));
    }
}

Register a data table

In the PAI-Rec console, go to Metric Management > Data Registration. Associate a MaxCompute table with your A/B testing instance. Set Data Table Name to a name that reflects the business context.

Select a table that contains the required fields. For field requirements, see Data registration and field configuration.

If your MaxCompute project does not have a table with all required fields, create one:

1. Log on to the MaxCompute console and create a data table.

2. In DataWorks, create a source table for A/B testing experiment reports in MaxCompute.

After registration, the table appears in the data table list. Click View Fields in the Actions column to inspect or edit field mappings.

Metric types

PAI-Rec supports three categories of metrics:

In addition, distinguish between:

• Key metrics (KPIs): the primary outcomes the experiment is designed to improve

• Must-see metrics: metrics that must be observed during a test; the capability being tested cannot affect them adversely

Configure metrics

On the Metric Configurations page, click Metrics. Set the Recommendation Scenario and Metric Timeliness parameters.

Single-dimension metrics are computed from a single aggregation: count, unique count, sum, or average. For example, daily clicks and daily visits are single-dimension metrics.

In the Metric Definition parameter:

• Page Views: counts every page visit

• Unique Visitors: counts each user once, on first visit

Derived metrics combine multiple single-dimension metrics. For example:

CTR = Daily clicks / Daily visits

Add daily clicks and daily visits as single-dimension metrics first, then define CTR as a derived metric using the formula above.

Configure metric groups

Group related metrics together to calculate and compare them as a set.

1. On the Metric Groups tab, create a metric group and set the Metric Selection parameter to include the metrics you want.

2. Click Calculate in the Actions column to trigger metric computation.

Calculate metrics

On the Metric Groups tab of the Metric Configurations page, click Calculate in the Actions column. Select the metrics to compute on the panel that appears.

Track job progress on the Jobs page. When all jobs complete, the status changes to Succeeded.

View reports

On the Performance Reports page under Experiment Platform, configure the parameters and click Start Analysis.

• Detail Data: shows the metric differences between the baseline experiment and normal experiments

• Trend Analysis: shows how selected metrics change over the experiment period