All Products
Search
Document Center

A/B test report

Last Updated: Sep 09, 2021

Description

OpenSearch provides five categories of statistical metrics for A/B tests: core metrics, traffic metrics, behavioral metrics, transaction metrics, and user analysis metrics.

Core metrics

Metric

Definition

Calculation method

Dependency

Description

Search PV

The number of successful search requests. PV is short for page view. Search requests include paging requests.

Counts each search request for which a success response is received.

Calculates the number of successful search requests on the current day.

Search UV

The number of users who have visited search results pages. UV is short for unique visitor.

Counts each user only once regardless of how many times the user has visited search results pages.

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation.

Calculates the number of users who have sent search requests on the current day.

Page turning rate

The ratio of paging requests to all search requests.

Divides the number of paging requests by Search PV.

Calculates the ratio of paging requests to all search requests to assess the effects of search retrieval and search results sorting.

No result rate

The ratio of search requests whose results contain no items to all search requests.

Divides the number of search requests whose results contain no items by Search PV.

Calculates the ratio of search requests whose results contain no items to all search requests to assess the effect and content diversity of search retrieval.

PV-CTR

The click-through rate (CTR) of search results.

Divides IPV by Search PV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of search results to assess the effects of search retrieval and search results sorting.

UV-CTR

The CTR of users.

Divides IPV_UV by Search UV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of users to assess the effects of search retrieval and search results sorting.

TOP5 PV-CTR

The CTR of the top five items in search results. These items are the first five items on the first page of search results.

Divides the number of clicks on the top five items in search results by Search PV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of the top five items in search results to assess the effect of search results sorting.

Search PV per capita

The average number of successful search requests per user.

Divides Search PV by Search UV.

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation.

Calculates the average number of successful search requests per user. More search requests indicate that users are interested in searching but often need to search for many times to obtain the expected results.

Next day retention rate

The ratio of users who have sent search requests for two consecutive days to users who sent search requests on the previous day.

Divides the intersection of Search UV (t-1) and Search UV (t) by Search UV (t-1).

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation.

Indicates user stickiness.

Traffic metrics

Metric

Definition

Calculation method

Dependency

Description

Search PV

The number of successful search requests. Search requests include paging requests.

Counts each search request for which a success response is received.

Calculates the number of successful search requests on the current day.

Search UV

The number of users who have visited search results pages.

Counts each user only once regardless of how many times the user has visited search results pages.

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation.

Calculates the number of users who have sent search requests on the current day.

Search PV per capita

The average number of successful search requests per user.

Divides Search PV by Search UV.

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation.

Calculates the average number of successful search requests per user. More search requests indicate that users are interested in searching but often need to search for many times to obtain the expected results.

Item impressions

The number of item exposures. Items refer to commodities or content on search results pages. If an item on the current page appears on the new page after a paging request or loading process, the second appearance of the item is counted as a new item exposure.

If the data collection feature is enabled and data about item exposures is uploaded, each item exposure is counted for this metric. Otherwise, the total number of items in search results is counted for this metric.

Depends on the data collection feature and the uploaded behavioral data about item exposures for higher accuracy. If the data collection feature is disabled and no behavioral data about item exposures is uploaded, the accuracy is low.

Calculates the number of item exposures in search results on the current day.

Query number

The total number of queries from all users after deduplication by user. For each user, the same query that the user performed multiple times is counted only once.

Aggregates the numbers of deduplicated queries from each user. The same query that is performed by two users is counted twice.

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation. If a search request contains the raw_query field, the raw query is used. Otherwise, the value for the default index is used as the query. If no default index is specified, the query is empty.

Calculates the total number of queries from all users after deduplication by user. Deduplication prevents repeated counting of paging requests and repeated searches.

Number of queries per capita

The average number of deduplicated queries per user.

Divides Query number by Search UV.

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation.

Calculates the average number of deduplicated queries per user. Deduplication prevents repeated counting of paging requests and repeated searches.

Number of independent queries

The number of distinct queries from all users.

The number of queries from all users after deduplication. The same query that is performed by multiple users is counted only once.

Depends on the raw_query field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation. This dependency is optional. If a search request contains the raw_query field, the raw query is used. Otherwise, the value for the default index is used as the query. If no default index is specified, the query is empty.

Indicates the diversity of queries.

Page turning rate

The ratio of paging requests to all search requests.

Divides the number of paging requests by Search PV.

Calculates the ratio of paging requests to all search requests to assess the effects of search retrieval and search results sorting.

No result rate

The ratio of search requests whose results contain no items to all search requests.

Divides the number of search requests whose results contain no items by Search PV.

Calculates the ratio of search requests whose results contain no items to all search requests to assess the effect and content diversity of search retrieval.

Low result rate

The ratio of search requests whose results contain no more than five items to all search requests.

Divides the number of search requests whose results contain no more than five items by Search PV.

Calculates the ratio of search requests whose results contain less than five items to all search requests to assess the effect and content diversity of search retrieval.

Request error times

The number of times when a search request is sent but an error response is received due to reasons such as throttling.

Counts each failed request.

Indicates whether you need to adjust the application, for example, scale out the application.

Request error rate

The ratio of failed search requests to all search requests. The requests failed due to reasons such as throttling.

Divides Request error times by the sum of Search PV and Request error times.

Indicates whether you need to adjust the application, for example, scale out the application.

Behavioral metrics

Metric

Definition

Calculation method

Dependency

Description

IPV

The number of clicks on items in search results.

Counts each click on an item in search results.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the number of clicks on items in search results.

IPV_UV

The number of users who have clicked items on search results pages.

Counts each user only once regardless of how many times the user has clicked items in search results.

Depends on the data collection feature and the uploaded behavioral data about user clicks. The user_id field must be specified in the data that is uploaded.

Calculates the number of users who have clicked items in search results.

IPV per capita

The average number of clicks on items in search results per user.

Divides IPV by Search UV.

Depends on the data collection feature and the uploaded behavioral data about user clicks. The user_id field must be specified in the data that is uploaded.

Calculates the average number of clicks on items in search results per user.

PV-CTR

The CTR of search results.

Divides IPV by Search PV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of search results to assess the effects of search retrieval and search results sorting.

UV-CTR

The CTR of users.

Divides IPV_UV by Search UV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of users to assess the effects of search retrieval and search results sorting.

Item-CTR

The CTR of items. Items refer to commodities or content on search results pages.

Divides IPV by Item impressions.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of items to assess the effects of search retrieval and search results sorting.

Percentage of clicked PV

The ratio of search requests whose results are clicked by users to all search requests.

Divides the number of search requests whose results are clicked by users by Search PV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the ratio of search requests whose results are clicked by users to all search requests to assess the effects of search retrieval and search results sorting.

TOP3 PV-CTR

The CTR of the top three items in search results. These items are the first three items on the first page of search results.

Divides the number of clicks on the top three items in search results by Search PV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of the top three items in search results to assess the effect of search results sorting.

TOP5 PV-CTR

The CTR of the top five items in search results. These items are the first five items on the first page of search results.

Divides the number of clicks on the top five items in search results by Search PV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of the top five items in search results to assess the effect of search results sorting.

TOP10 PV-CTR

The CTR of the top ten items in search results. These items are the first ten items on the first page of search results.

Divides the number of clicks on the top ten items in search results by Search PV.

Depends on the data collection feature and the uploaded behavioral data about user clicks.

Calculates the CTR of the top ten items in search results to assess the effect of search results sorting.

Number of collections

The number of times when users add items from search results to favorites.

Counts each time when a user adds an item from search results to favorites.

Depends on the data collection feature and the uploaded behavioral data about adding items to favorites. The value of the bhv_type field must be "collect" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to add items to favorites.

Favorite conversion rate

The ratio of times when users add items from search results to favorites to all search requests.

Divides Number of collections by Search PV.

Depends on the data collection feature and the uploaded behavioral data about adding items to favorites. The value of the bhv_type field must be "collect" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to add items to favorites.

Additional purchases

The number of times when users add items from search results to the shopping cart.

Counts each time when a user adds an item from search results to the shopping cart.

Depends on the data collection feature and the uploaded behavioral data about adding items to the shopping cart, bookshelf, or playlist. The value of the bhv_type field must be "cart" in the data that is uploaded.

Assess the effects of search retrieval, search results sorting, and search results on guiding users to add items to the shopping cart.

Add-on conversion rate

The ratio of times when users add items from search results to the shopping cart to all search requests.

Divides Additional purchases by Search PV.

Depends on the data collection feature and the uploaded behavioral data about adding items to the shopping cart, bookshelf, or playlist. The value of the bhv_type field must be "cart" in the data that is uploaded.

Assess the effects of search retrieval, search results sorting, and search results on guiding users to add items to the shopping cart.

Likes

The number of times when users like items in search results.

Counts each time when a user likes an item in search results.

Depends on the data collection feature and the uploaded behavioral data about liking items. The value of the bhv_type field must be "like" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to like items in search results.

Like conversion rate

The ratio of times when users like items in search results to all search requests.

Divides Likes by Search PV.

Depends on the data collection feature and the uploaded behavioral data about liking items. The value of the bhv_type field must be "like" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to like items in search results.

Number of comments

The number of times when users post comments on items in search results.

Counts each time when a user posts comments on an item in search results.

Depends on the data collection feature and the uploaded behavioral data about posting comments. The value of the bhv_type field must be "comment" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to post comments.

Comment conversion rate

The ratio of times when users post comments on items in search results to all search requests.

Divides Number of comments by Search PV.

Depends on the data collection feature and the uploaded behavioral data about posting comments. The value of the bhv_type field must be "comment" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to post comments.

Transaction metrics

Metric

Definition

Calculation method

Dependency

Description

Search guide GMV

The gross merchandise volume (GMV) of purchases that users make based on search results.

Aggregates the transaction amounts.

Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be "buy" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases.

Search guide deal UV

The number of users who have made purchases based on search results.

Counts each user only once regardless of how many purchases the user has made based on search results.

Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be "buy" in the data that is uploaded. The user_id field must be specified in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases.

Search guide customer unit price

The average transaction amount per user among users who have made purchases based on search results.

Divides Search guide GMV by Search guide deal UV.

Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be "buy" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases.

Order conversion rate

The ratio of users who have made purchases based on search results to all users who have visited search results pages.

Divides Search guide deal UV by Search UV.

Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be "buy" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases.

Click-through rate

The ratio of users who have made purchases after they clicked items in search results to all users who have clicked items in search results.

Divides Search guide deal UV by IPV_UV.

Depends on the data collection feature and the uploaded behavioral data about user purchases. The value of the bhv_type field must be "buy" in the data that is uploaded.

Assesses the effects of search retrieval, search results sorting, and search results on guiding users to make purchases.

User analysis metrics

Metric

Definition

Calculation method

Dependency

Description

Next day retention rate

The ratio of users who have sent search requests for two consecutive days to users who sent search requests on the previous day.

Divides the intersection of Search UV (t-1) and Search UV (t) by Search UV (t-1).

Depends on the user_id field that is specified in search requests when users send requests by using OpenSearch SDK or calling the API operation.

Indicates user stickiness.

Lost user rate

The ratio of users who did not click items in search results to all users who have visited search results pages.

Divides the difference between UV and IPV_UV by Search UV.

Depends on the data collection feature and the uploaded behavioral data about user clicks. The user_id field must be specified in the data that is uploaded.

Assesses the effects of search retrieval and search results sorting.

Usage notes

  1. When the A/B test report displays metrics for a day, the report displays the metrics for the day before the current day by default. You can set the date picker control to another day as needed. After you upload behavioral data, the A/B test report based on the uploaded data is available the next day.

  2. The A/B test report provides line charts to display the trends of metrics in the last seven days. You can set the date picker control to another time period as needed. You can view the line chart of a metric by clicking the corresponding icon.

  3. To view a line chart, perform the following steps: In the left-side navigation pane, choose Report Statistics > A/B test report. Select an application from the Application Name drop-down list, an experiment group, and then an experiment. Click a metric category tab. Then, set the date picker control to a time period as needed. 1