A/B Testing Concepts and Traffic Splitting for AI Models - Platform for AI

Limits

Currently, ABTest:

Is available only in the China (Beijing), China (Shanghai), China (Hangzhou), and China (Shenzhen) regions.
Does not support role-based SSO access. Access ABTest only as a RAM user.

Billing

ABTest is in invitational preview and is free to use. However, it uses other Alibaba Cloud products, such as Elastic Algorithm Service (EAS) and MaxCompute, which are billed separately. For more information, see EAS billing and MaxCompute billing overview.

Terms

Experiment management

Experiment: A combination of different experiment versions for A/B testing, such as AA, AB, and ABC. Each experiment receives traffic from an experiment layer. A layer can contain multiple experiments with mutually exclusive traffic. An experiment manages parameter settings that control program execution flow. Traffic for each version is randomly divided to enable comparison of different parameters.
Experiment project: An aggregation of related business logic. For example, in an LLM Agent scenario, services in similar business scenarios can be grouped into a single project.
Experiment domain: A collection of traffic filtered by conditions based on business properties or randomly allocated. If business logic determines traffic division for the domain, implement custom division logic.
Experiment layer: An experiment domain contains one or more experiment layers. Traffic in each layer is orthogonal (mutually independent) and contains all domain traffic.

Traffic management

Audience: A collection of specific traffic IDs, such as user UIDs, that are part of an experiment.

Metric management

Metric: An indicator, such as performance or quality of service, used to evaluate experiment outcomes.
Data table: The data source and related fields required to calculate experiment metrics.

Global configuration

Publish management: Promotes and applies the parameters of a more effective experiment version.

Architecture

Overall architecture

Use the Alibaba Cloud ABTest web console to configure experiments and metrics.
ABTest provides Go and Java SDKs for server-side applications. The SDK pulls experiment metadata, splits traffic based on access context, and runs relevant business logic.
Register a MaxCompute log table as the source data table, which the system then registers with the ABTest server. When an application generates behavior logs, instrumentation sends data back to the MaxCompute log table. If the log table generates content in near-real-time, configure experiment metrics in the console to generate both hourly and daily metrics. Metrics are stored in the Hologres instance for the ABTest service.
View experiment report data in the ABTest web console.

Experiment evolution

Single-layer experiment: After creating a project, the system automatically creates a default domain and layer. Create an experiment on the default layer. The experiment can use all layer traffic or a portion through random allocation. An experiment includes multiple versions. Divide total traffic among versions by setting a traffic percentage for each.
Multilayer experiment: A single-layer experiment can be expanded into multiple layers. Traffic on each layer is orthogonal, allowing experiments to be set up on each layer.
Layer and domain combination: A layer can contain multiple domains, and a domain can also have multiple layers, as shown in the figure. In a multilayer traffic experiment, run a cross-layer experiment or an experiment on a single layer. Configure the following settings:

Permissions

Grant RAM user permissions to access ABTest

After granting management permissions for ABTest to a RAM user, the user has full access to all ABTest features.

Log on to the RAM console with your root account.
Create a custom policy. In the script editor, copy the following content. Set Policy Name to pai_abtest_full_access. For more information, see Create a custom policy.
```
{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "paiabtest:*",
      "Resource": "*"
    }
  ]
}
```
On the authorization page, grant pai_abtest_full_access permission to the RAM user. For more information, see Manage permissions for a RAM user.

Grant ABTest permissions to access other cloud products

ABTest requires authorization through a service-linked role (SLR). The role name is AliyunServiceRoleForPAIABTest. The policy contains these permissions.

{
  "Version": "1",
  "Statement": [
    {
      "Action": "ram:DeleteServiceLinkedRole",
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "ram:ServiceName": "abtest.pai.aliyuncs.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "odps:ActOnBehalfOfAnotherUser",
        "odps:ListProjects",
        "odps:ListTables"
      ],
      "Resource": "acs:odps:*:*:users/*"
    }
  ]
}