All Products
Search
Document Center

A/B tests

Last Updated: Sep 09, 2021

To help you use and debug various algorithms in OpenSearch, OpenSearch provides the A/B test feature. Before you use a feature in your online business, you can run an A/B test to test the feature by routing a specified proportion of traffic of queries to the specified test group. This helps avoid negative impact on your online business in scenarios in which you use a feature without testing it. The A/B test feature allows you to test the effect of query analysis, rough sort, and fine sort.

Use the A/B test feature

1. Configure an A/B test

To obtain more comprehensive test metrics, we recommend that you enable the feature to collect click behavioral data before you configure an A/B test. If you use the A/B test feature for the first time, you must perform the following four steps to configure an A/B test:

1.Go to the Create Test Group page.

Log on to the OpenSearch console. In the left-side navigation pane, choose Feature Extensions > A/B Test. On the A/B Test page, click Create.1

2.Create a test group.

You can create a test group to test the effects of query analysis, rough sort, fine sort, and category prediction. Test Group Name: the name of the test group to be created. The name is for display purpose only. You can specify the name as needed. The name of a test group cannot exceed 30 characters in length.2

3.Create a test.

After you specify the name of the test group and click Next, you can create a test by clicking Create Test in the Configure Test step. You can create up to 20 tests in a test group, excluding the baseline test that uses the default online configuration. However, only a maximum of 10 tests can take effect at the same time.3

3.1 Test Name: the name of the test to be created. You must specify the test name. The name of a test cannot exceed 30 characters in length. 3.13.2 Configure the test.

You can specify the query analysis, rough sort, and fine sort that you want to test. You can select only existing query analysis rules, rough sorts, and fine sorts.

When you specify the query analysis for a test in the Query Analysis Types and Rules column:

  • If you select Custom from the drop-down list, you must select a query analysis rule that you create for the application.

  • If you select Use Default Online Configuration from the drop-down list, the default online configuration is used for the test.

The preceding logic also applies when you specify the rough sort and fine sort for a test in the Sort Policy Types and Policies column.

You may unintentionally modify a sort policy that is being configured. If you select a sort policy that is being configured, the effect of the A/B test is affected. To resolve the issue, you are allowed to select only sort policies that are published. In addition, after you select a sort policy, you are not allowed to modify it.3.2

3.3 Test Traffic: The minimum traffic proportion that you can set for a test is 1%. In the same scenario, the sum of the traffic proportions that you set for all online tests in a test group cannot exceed 100%.3.3

4.Complete the creation of the test group.

After you create and configure the tests for the test group, click Next. The "The test group is created" message appears in the Complete step. Click Complete to return to the A/B Test page. 4The following figure shows that a created test group is in the Pending state.4.2

5.Start testing.

After you create a test group, you can click Start Testing in the Actions column of the test group to start testing. When the test is started, the status of the test group becomes Testing.5

6.Use the A/B test for online search queries.

After the A/B test feature is enabled and the test group is created and configured, if you want to use the A/B test for online search queries, you must specify the abtest parameter in a query. The abtest parameter consists of two parameters: scene_tag and flow_divider.6

The following code is a URL that is used for a search query:

/v3/openapi/apps/160029126/search?query=query=default:'Shenzhen'&&config=start:0,hit:10,format:fulljson&abtest=scene_tag:test_1,flow_divider:123456
  • scene_tag: the name of the test group. Originally, scene_tag indicates the tag of the test scene. You can set the value of this parameter to the name of a test group that you create in the OpenSearch console as shown in the preceding figure. This way, the test traffic that you set for the tests in the test group is sent to the corresponding tests.

  • flow_divider: You must specify this parameter. The backend system performs the hash operation on the value of this parameter to allocate query traffic to different tests in the test group. The backend system allocates the traffic based on the proportions that you configure for each test in the OpenSearch console. We recommend that you set the flow_divider parameter to the ID of a user. You can also set the parameter to the ID or IP address of the user device.

Note:

  • When you access OpenSearch by using OpenSearch SDKs, if the values of the scene_tag and flow_divider parameters do not contain spaces or punctuations, you can directly call the corresponding methods without encoding the values of the scene_tag and flow_divider parameters. If the values of the scene_tag and flow_divider parameters contain spaces or punctuations, you must first encode the values of the scene_tag and flow_divider parameters. For more information, see Practical examples.

  • If you access OpenSearch by using API operations, the values of the scene_tag and flow_divider parameters must be URL-encoded. The value of the abtest parameter is passed to OpenSearch in the following format: abtest=urlencode(scene_tag:urlencode(\$scene),flow_divider:urlencode(\$value)). In the format, urlencode is a function that is used to encode URLs.

  • For more information, see FAQ about A/B tests.

2. Manage test groups and tests.

Manage test groups.

After a test group is created, it is displayed in the test group list on the A/B Test page.10You can perform the following basic operations on test groups that you create:

1.Start testing.

You can start testing for the test group whose status is Pending or Stopped.

2.Stop testing.

You can stop testing for the test group whose status is Testing.

3.Delete a test group.

You can delete a test group on the A/B Test page.

Manage a test.

After a test group is created, it is displayed in test group list on the A/B Test page.11To view the details of a test group, you can click Details of the test group in the Actions column. On the Test Group Details page, the Test List section lists the name, query analysis types and rules, sort policy types and policies, traffic of the tests in the test group, and the operations that you can perform on tests. 12You can perform the following basic operations on a test:

  1. Edit a test: You can edit the name, configuration, and traffic of a test.

  2. Delete a test: If you delete a test, the system no longer retains its configurations. The test that you delete no longer takes effect online.

  3. Configure a whitelist:

OpenSearch provides the whitelist feature, which allows you to add a specified value of the flow_divider parameter to a specified test. This way, you can view the search effect of the test.13

3. View the details of an A/B test group

Behavioral data status

After you create a test group, you can click Details of the test group in the Actions column on the A/B Test page. On the Test Group Details page, you can view the behavioral data status. The behavioral data may have the following status:

  • Not activated: No behavioral data is uploaded for the application. For more information, see Data collection 2.0.

  • Activated with no data: The data collection feature is enabled for the application. However, no data is received by the application.

  • Abnormal (Unavailable): The behavioral data is found to be untrustworthy based on the quality check.

Test group status

After you create a test group, you can click Details of the test group in the Actions column on the A/B Test page. On the Test Group Details page, you can view the status of the test group. The test group may have the following status:

  • Pending: the status of the test group. This status indicates that a test was started or stopped for the test group.

  • Testing: indicates that a test is started for the test group. The number of days that elapsed since the day when the test is started is displayed.

  • Stopped: indicates that a test is stopped for the test group. The accumulated test time is calculated from the time when the test is started for the first time to the time when the test is finally stopped. Only the number of days on which the test is actually running is accumulated.

Data statistics

To view the data statistics of a test group, find a test group that you want to view on the A/B Test page, and click the Details of the test group in the Actions column to go to the Test Group Details page. On the Data Statistics section, you can view the data report of the A/B test. Alternatively, you can choose Report Statistics > A/B test report in the left-side navigation pane to go to the A/B test report page. In the OpenSearch console, you can view the data statistics of a test group one day after the data is generated. After the OpenSearch console is upgraded, core metrics comparison page and specific data table are merged. You can select the metrics that you want to view from the Metric Type drop-down list on the Test Group Details page of a test group. Instead of the original data form, the core data metrics are displayed in a line chart. By default, the data metrics in the last day are displayed. On the Data Statistics section of the Test Group Details page, you can select multiple tests whose data you want to compare from the Test drop-down list. Available core metrics include page views (PVs), user views (UVs), zero-result rate, average PVs, exposure times, search queries, and average search queries.

  • Note: If you start an A/B test on a day, you can view the data that is generated for the test after 08:00. the next day. If you stop the test on the day, you can view the data that is generated during the test.

Practical examples

Assume that you want to enable the OpenSearch service for an e-commerce product to achieve two objectives.

  • First objective: Query the traffic of the search requests that are sent from customers based on the product keyword search. Example:

query=config=format:fulljson&&query=default:'baby milk powder'&&sort=price

  • Second objective: Query the call traffic from other internal business. Example:

query=config=format:fulljson&&query=cat_id:'1'|'2'|'3'&&sort=timestamp

For your first objective, if you want to perform an A/B test based on the member ID to compare the effects of several sort expressions, category prediction models, or query analysis rules, you can perform the following steps:

1.Create a test group and required tests by using the A/B test feature in the OpenSearch console. In the Basic Information step on the Create Test Group page, set the Test Group Name to user_search.14

2.Set the abtest parameter in the search query. The Test Group Name parameter is set to user_search in the OpenSearch console. Therefore, you must set the scene_tag parameter to user_search and the flow_divider parameter to xxxx, which is the value of the member ID, in the search query.

2.1 Use SDKs. The following code shows how to use Java SDK to set the search query. The use of PHP SDK is similar to that of Java SDK:15aliyun-sdk-opensearch-3.4.1 (Java) and opensearch-sdk-php-3.2.1 (PHP).

2.2 Use APIs.

i. query=config=format:fulljson&&query=default:'baby milk'&&sort=-price&abtest=scene_tag:user_search,flow_divider:%e5%bc%a0%e4%b8%89

Note

Note: The values of the scene_tag and flow_divider sub-parameters of the abtest parameter are URL-encoded.

ii. Perform URL encoding on the value of each parameter, such as query, sort, and abtest, in a request.

query=config%3dformat%3afulljson&&query%3ddefault%3a%27%e5%ae%9d%e5%ae%9d%e5%a5%b6%e7%b2%8

3.After you complete the preceding configurations, you can perform the A/B test by routing the traffic of search queries from customers based on the member ID to the test group.

Business operation report

User interfaces

To go to the A/B test report page, perform the following steps: In the left-side navigation pane in the OpenSearch console, choose Feature Extensions > A/B Test. Find the test that you want to view, and click Statistical report in the Actions column.16

You can also choose Report Statistics > A/B test report to go to the A/B test report page.17

Note: For more information about metrics that you can view on the A/B test report page, see A/B test reports.