ABTest overview - Platform For AI - Alibaba Cloud Documentation Center

ABTest is a platform for running controlled experiments on model services and online performance optimization — helping you compare experiment versions, analyze metrics, and promote the best-performing configuration to production.

Limits

Currently, ABTest:

Is available only in the China (Beijing), China (Shanghai), China (Hangzhou), and China (Shenzhen) regions.
Does not support role-based SSO access. Access ABTest only as a RAM user.

Billing

ABTest is in invitational preview and is free to use. However, it uses other Alibaba Cloud products, such as Elastic Algorithm Service (EAS) and MaxCompute, which are billed separately. For more information, see EAS billing and MaxCompute billing overview.

Terms

Experiment management

Experiment: A controlled comparison of different experiment versions (AA, AB, or ABC). Each experiment draws traffic from an experiment layer. Within a layer, multiple experiments run with mutually exclusive traffic, and each version receives a randomly allocated share for parameter comparison. An experiment manages parameter settings that control program execution flow.
Experiment project: A logical grouping of related services. For example, services in similar LLM Agent scenarios can share a single project.
Experiment domain: A pool of traffic filtered by conditions based on business properties or random allocation. When business logic governs traffic division, implement custom splitting logic for the domain.
Experiment layer: One or more layers within an experiment domain. Traffic in each layer is orthogonal — each layer carries all domain traffic independently, so experiments on different layers do not interfere.

Traffic management

Audience: A defined set of traffic IDs — such as user UIDs — included in an experiment.

Metric management

Metric: An indicator — such as performance or quality of service — used to evaluate experiment outcomes.
Data table: The data source and associated fields used to calculate experiment metrics.

Global configuration

Publish management: Promotes and applies the parameters of a more effective experiment version.

Architecture

Overall architecture

Use the Alibaba Cloud ABTest web console to configure experiments and metrics.
ABTest provides Go and Java SDKs for server-side applications. Each SDK pulls experiment metadata, splits traffic based on access context, and executes the relevant business logic.
Register a MaxCompute log table as the source data table, which the system then registers with the ABTest server. When your application generates behavior logs, instrumentation sends the data back to that table. If the log table produces data in near-real-time, configure experiment metrics in the console to generate hourly and daily metric reports. Metrics are stored in the Hologres instance for the ABTest service.
View experiment report data in the ABTest web console.

Experiment evolution

Single-layer experiment: When you create a project, ABTest automatically creates a default domain and layer. Create an experiment on that default layer using all available traffic or a randomly allocated subset. An experiment includes multiple versions. Divide the total traffic among experiment versions by assigning a percentage to each.
Multilayer experiment: Expand a single-layer experiment into multiple layers. Traffic on each layer is orthogonal, so each layer can run independent experiments without interference.
Layer and domain combination: A layer can contain multiple domains, and a domain can span multiple layers. Use this configuration to run cross-layer experiments or to isolate an experiment to a single layer.

Permissions

Grant RAM user permissions to access ABTest

Granting management permissions for ABTest to a RAM user gives that user full access to all ABTest features.

Log on to the RAM console with your root account.
Create a custom policy. In the script editor, paste the following content and set Policy Name to pai_abtest_full_access. For more information, see Create a custom policy.
```
{
  "Version": "1",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "paiabtest:*",
      "Resource": "*"
    }
  ]
}
```
On the authorization page, grant pai_abtest_full_access to the RAM user. For more information, see Manage permissions for a RAM user.

Grant ABTest permissions to access other cloud products

ABTest uses a service-linked role (SLR) named AliyunServiceRoleForPAIABTest to access other cloud products. The role includes the following permissions.

{
  "Version": "1",
  "Statement": [
    {
      "Action": "ram:DeleteServiceLinkedRole",
      "Resource": "*",
      "Effect": "Allow",
      "Condition": {
        "StringEquals": {
          "ram:ServiceName": "abtest.pai.aliyuncs.com"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "odps:ActOnBehalfOfAnotherUser",
        "odps:ListProjects",
        "odps:ListTables"
      ],
      "Resource": "acs:odps:*:*:users/*"
    }
  ]
}