Automatic service stress testing - Machine Learning Platform for AI

eas-benchmark is a distributed general-purpose stress testing tool provided by Elastic Algorithm Service (EAS). eas-benchmark supports dynamic stress testing, allows you to view real-time monitoring data that is accurate to seconds, and can generate a visualized report after a stress testing task is completed. You can use eas-benchmark to create stress testing tasks for services that are deployed in EAS. This helps you learn the performance limit of these services. This topic describes how to use eas-benchmark to create a stress testing task and how to manage stress testing tasks.

Prerequisites

A model service is deployed. For more information, see Model service deployment by using the PAI console and Machine Learning Designer.

Getting started

You can log on to the EASCMD client and run the bench commands to create and manage stress testing tasks. For more information about how to create and manage stress testing tasks, see Create a stress testing task and Manage stress testing tasks. For more information about how to log on to the EASCMD client, see Download the EASCMD client and complete user authentication.

Create a stress testing task

Use the EASCMD client to create a stress testing task

Run the bench create command to create a stress testing task. After the stress testing task is created, you can view the real-time monitoring data by using the returned URL. Command:

eascmd bench create [bench_desc_json]

bench_desc_json is the JSON file that stores stress testing-related information. The following code block shows an example:

{
    "service": {
        "serviceName": "xgb_test"
    },
    "data": {
        "content": "W1sxLDAsMCwwLDEsMSwwLDEsMCwxLDEsMCwwLDEsMCwxLDAsMSwwLDAsMSwxLDEsMCwxLDEsMCwwLDAsMSwxLDEsMCwxLDEsMSwxLDAsMSwxLDEsMCwxLDAsMCwwLDEsMSwwLDAsMCwxLDAsMSwwLDEsMCwwLDEsMCwwLDEsMCwxLDAsMCwxLDAsMCwwLDAsMSwwLDEsMCwxLDAsMCwxLDEsMSwwLDAsMSwwLDAsMCwwLDEsMSwxLDAsMSwxLDAsMCwxLDAsMSwwLDEsMSwxLDEsMCwxLDAsMCwxLDEsMSwxLDAsMCwwLDEsMSwwXV0K"
    }
}

The path parameter specifies the path of the Object Storage Service (OSS) files that store the stress testing data. The following code block shows an example. If you want to submit multiple stress testing requests, you can package the requests into a ZIP file and set path to the path of the ZIP file in the OSS://XX.zip format.

{
    "service": {
        "serviceName": "xgb_test"
    },
    "data": {
        "path": "OSS://examplebucket/test1.bin,OSS://examplebucket/test2.bin"
    }
}

The following table describes the parameters in the JSON file.


Section	Parameter	Required	Description
service	serviceName	Yes	The name of the service that you want to test.
data	content	No	The content of a stress testing request, which is a Base64-encoded string. If you want to submit multiple requests, specify the path parameter.
	path	No	The path where the stress testing data is stored. You can specify an HTTP path or OSS path. You can specify multiple paths and separate them with commas (,). You can also package multiple stress testing files into a ZIP file and then specify the path of the ZIP file. Note You do not need to use base64 to encode the files that store the stress testing data.
	multiLine	No	Specify whether to separate the stress testing data by row. Valid values: true and false. Default value: false. If you set the parameter to true, the downloaded data is parsed by row.
http	headers	No	The HTTP request headers. The parameter is of LIST type. Example: ["Authorization:aaa", "Content-Type:text"].
http	timeout	No	The HTTP request latency in milliseconds. Default value: 20000.
optional	mode	No	The stress testing mode. Valid values: auto: automatic scan: periodical manual: manual Default value: auto. For more information about the stress testing modes, see Introduction to the stress testing modes.
	duration	No	The stress testing duration in seconds. Default value: 600. Maximum value: 1200.
	agentCount	No	This parameter is available when you set the stress testing mode to manual. This parameter specifies the number of agent workers. The loads increase with the number of agent workers. Default value: 1.
	concurrency	No	This parameter is available when you set the stress testing mode to manual. This parameter specifies the concurrency of each agent worker. The loads increase with the concurrency. Default value: 2. When you want to increase the loads of the service, increase the concurrency value first. When the loads stop increasing with the concurrency, try to add more agent workers.
	adjustInterval	No	This parameter is available when you set the stress testing mode to scan. This parameter specifies the interval at which stress tests are automatically performed in seconds. Default value: 60.
	minQPS	No	This parameter is available when you set the stress testing mode to scan. This parameter specifies the initial QPS of automatic stress testing. Default value: 100.
	maxQPS	No	This parameter is available when you set the stress testing mode to scan or auto. This parameter specifies the maximum QPS value.
	maxRT	No	This parameter is available when you set the stress testing mode to scan or auto. This parameter specifies the maximum response time (TP99). When the actual response time exceeds the specified upper limit, the QPS is automatically adjusted until the response time drops below the upper limit.
	qpsGrowthDelta	No	This parameter is available when you set the stress testing mode to scan. This parameter specifies the QPS increment each time. Default value: 50.
	faultTolerate	No	This parameter is available when you set the stress testing mode to scan or auto. This parameter specifies the highest request error rate that can be tolerated, excluding 200 errors. If you set the parameter to 0.01, request errors are handled when more than 1% of the requests counter an error. The default value is 0.001, which means that the highest request error rate that can be tolerated is 0.1%.
	faultAction	No	This parameter is available when you set the stress testing mode to scan or auto. When the request error rate exceeds the threshold specified by the faultTolerate parameter, eas-benchmark performs one of the following operations: stop: maintains the current QPS and stops increasing the loads. revise: dynamically adjusts the QPS until the request error rate meets the requirement. Default value: revise.

The following code block shows an example. In this example, the JSON file that stores the stress testing task information is named bench_xgb.json.

$ eascmd bench create bench_xgb.json

The system displays information similar to the following output:

[RequestId]: DE240637-4976-59AF-A28C-BAA55C0AA259
[OK] Task [benchmark-xgb-test-b514] is creating
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 1/1]: Benchmark task is Running
[OK] Benchmark task is Running
[OK] Click the link http://127.0.0.1:18222/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C.
Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command:
eascmd -c [config_file] bench visualize benchmark-xgb-test-b514

Use the console to create a stress testing task

All stress testing tasks created in the console use the auto mode. To create a stress testing task in the console, perform the following steps.

Go to the Elastic Algorithm Service page.
1. Log on to the Machine Learning Platform for AI console.
2. In the left-side navigation pane, click Workspaces. On the Workspace list page, click the name of the workspace that you want to manage.
3. In the left-side navigation pane, choose Model Deployment > Model Serving (EAS) to go to the Elastic Algorithm Service page.
On the Inference Service tab, click the service that you want to test to go to the service details page.
Perform the steps in the following figure to create a stress testing task.

In the Create Stress Testing Task panel, configure the parameters and click Create.


Parameter	Description
Testing Data	Select a data type, such as Single Data Entry, Data Address, or OSS Object. Note If you select Data Address or OSS Object, you can specify multiple file paths that are separated by commas (,). The uploaded stress testing data file is not segmented. The entire file is used to perform stress tests.
Duration (s)	Default value: 300. Unit: seconds.
Maximum QPS	The maximum QPS. Default value: 10000.
Maximum Response Time (ms)	The maximum response time. Unit: milliseconds. When the actual response time exceeds the specified upper limit, the QPS is automatically adjusted until the actual response time drops below the upper limit.

Manage stress testing tasks

Use the EASCMD client to manage stress testing tasks

View stress testing tasks

Run the bench list command to view the stress testing tasks that are created by the current user. Command:

eascmd bench ls

Example:

$ eascmd bench ls

The system displays information similar to the following output:

[RequestId]: 7F953F8E-8897-5785-808A-CA64830215F6
+-------------------------+--------------------------+-------------+----------------+---------+---------------------+
|        TASKNAME         |          TASKID          |   REGION    | AVAILABLEAGENT | STATUS  |     CREATETIME      |
+-------------------------+--------------------------+-------------+----------------+---------+---------------------+
| benchmark-xgb-test-7846 | eas-b-ql470xog6qeh251fdb | cn-shanghai |              0 | Stopped | 2022-06-17 17:58:01 |
| benchmark-xgb-test-b514 | eas-b-bdnzvwq0z0h3xqfk0s | cn-shanghai |              2 | Running | 2022-06-20 12:18:54 |
+-------------------------+--------------------------+-------------+----------------+---------+---------------------+

View the details of a stress testing task

Run the bench desc command to view the details of a stress testing task. Command:

eascmd bench desc [benchmark_task_name]

benchmark_task_name specifies the name of the stress testing task that you want to view. Example:

$ eascmd bench desc benchmark-demo-test-c7eb

The system displays information similar to the following output:

+----------------+------------------------------------------------------------------------------+
|     TaskName   | benchmark-xgb-test-b514                                                      |
|     TaskId     | eas-b-bdnzvwq0z0h3xqfk0s                                                     |
|    ServiceName | xgb_test                                                                     |
|         Region | cn-shanghai                                                                  |
|   DesiredAgent | 2                                                                            |
| AvailableAgent | 2                                                                            |
|         Status | Running                                                                      |
|        Message | Benchmark task is running                                                    |
|     CreateTime | 2021-10-20 12:38:35                                                          |
|     UpdateTime | 2021-10-20 12:38:45                                                          |
|         Config | {                                                                            |
|                |   "base": {                                                                  |
|                |     "agentCount": 2,                                                         |
|                |     "concurrency": 40,                                                       |
|                |     "duration": 1200,                                                        |
|                |     "requestCount":                                                          |
|                | 9223372036854776000,                                                         |
|                |   },                                                                         |
|                |  ...                                                                         |
|                | }                                                                            |
+----------------+------------------------------------------------------------------------------+

Enable real-time visualization for stress testing tasks
Run the bench visualize command to enable real-time visualization for stress testing tasks. This command launches a webserver that is accessible through 127.0.0.1 on your on-premises machine. The webserver provides a web page that displays the real-time monitoring data of stress testing tasks. Command:
```
eascmd bench visualize [benchmark_task_name]
```
Example:
```
$ eascmd bench visualize benchmark-xgb-test-b514
```
The system displays information similar to the following output:
```
[OK] Click the link http://127.0.0.1:18734/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C.
Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command:
eascmd -c [config_file] bench visualize benchmark-xgb-test-b514
```
You can enter http://127.0.0.1:18734/eas-benchmark/statsview into the address bar of your browser to view the real-time monitoring data.
Obtain the stress testing report
When the status of a stress testing task changes to Stopped, the task is completed. The stress testing report is saved in OSS. You can run the bench report command to obtain the report. Command:
```
eascmd  bench report [benchmark_task_name]
```
Example:
```
$ eascmd bench report benchmark-demo-test-c7eb
```
The system displays information similar to the following output:
```
[OK] Benchmark task benchmark-demo-test-c7eb report url: http://eas-benchmark.oss-cn-chengdu.aliyuncs.com/summary/benchmark-demo-test-c7eb-10004.html
```
Enter the link that follows url into the address bar of your browser to view the stress testing report, as shown in the following figure.

Dynamically modify the number of agent workers and concurrency

When the stress testing mode is manual, you can run the bench update command to modify the number of agent workers and concurrency. Command:

eascmd bench update [benchmark_task_name] -Doptional.concurrency=<attr_value> -Doptional.agentCount=<attr_value>

<attr_value> indicates the desired value. Example:

$ eascmd bench update benchmark-demo-b99c -Doptional.concurrency=2 -Doptional.agentCount=1

The system displays information similar to the following output:

[RequestId]: 9920C672-4D41-5CC4-8EC0-C690F76EB2BA
[OK] Running [TaskName: benchmark-demo-b99c, DesiredAgent:1, AvailableAgent: 1, Message: Benchmark task is Updating]
[OK] Benchmark task benchmark-demo-b99c was updated successfully

Stops a stress testing task
Run the bench stop command to stop a stress testing task. Command:
```
eascmd bench stop [benchmark_task_name]
```
Example:
```
$ eascmd bench stop benchmark-xgb-test-b514
```
The system displays information similar to the following output:
```
Are you sure to stop the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n]
[OK] Task [benchmark-xgb-test-b514] is stopping
[OK] [Agnet: 0/1]: Benchmark task is Running
[OK] [Agnet: 0/1]: Benchmark task is Stopped
[OK] Benchmark task is stopped
```
If real-time visualization is enabled for the task that you want to stop, the system generates the stress testing report on the terminal that performs real-time visualization. You can run the bench report command to view the detailed HTML report.

Start a stress testing task

Run the bench start command to start a stress testing task that has been stopped. Command:

eascmd bench start [benchmark_task_name]

Note Unlike the bench create command, this command restarts a stress testing task based on the latest configuration of the task.

Example:

$ eascmd bench start benchmark-xgb-test-b514

The system displays information similar to the following output:

Are you sure to start the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n]
[OK] Task [benchmark-xgb-test-b514] is starting
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 1/1]: Benchmark task is Running
[OK] Benchmark task is Running
[OK] Click the link http://127.0.0.1:18947/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C.
Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command:
eascmd -c [config_file] bench visualize benchmark-xgb-test-b514

Delete a stress testing task

After a stress testing task is completed, eas-benchmark retains the task based on the final state of the task. The following table describes the final states and corresponding retention periods.


Final state	Retention period
Stopped	48 hours
CreateFailed, UpdateFailed, Terminated, and Error	10 minutes

When the retention period ends, the system automatically deletes the task.

You can also run the bench delete command to delete the task. Command:

eascmd bench delete [benchmark_task_name]

Example:

$ eascmd bench delete benchmark-xgb-test-b514

The system displays information similar to the following output:

Are you sure to delete the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n]
[OK] Benchmark task benchmark-xgb-test-b514 is Deleting
[OK] Benchmark task was deleted successfully

Use the console to manage stress testing tasks

On the One-Click Stress Testing tab, you can view and manage the existing stress testing tasks. tasks

You can start, stop, or delete stress testing tasks with one click.
You can click the refresh button in the upper-right part of the page to refresh the status of the stress testing tasks.
When the Status column of a task displays Running, you can click View in the Actions column to view the real-time monitoring data of the task.
After a stress testing task is completed or you manually stop a task, the status of the task changes to Stopped. In this scenario, you can click View in the Actions column to view the stress testing report.

Introduction to the stress testing modes

eas-benchmark supports the following stress testing modes: auto, scan, and manual. You can specify the mode parameter in the optional section of the configuration file to specify the stress testing mode.

auto mode
The automatic stress testing mode. eas-benchmark automatically creates agent workers that are used for stress testing, sets the concurrency, and chooses the optimal algorithm to identify the performance upper limit of the service. In this mode, you need to only specify the service name and stress testing data in the configuration file and keep the default settings for other parameters. Sample configuration:
```
{
    "service": {
        "serviceName": "demo"
    },
    "data": {
        "path": "https://test-data.oss-cn-chengdu.aliyuncs.com/data/warmup.tf.bin"
    },
    "optional" {
        "maxQPS": 1000,
        "duration": 300
     }
}
```
scan mode
The periodical stress testing mode. In this mode, eas-benchmark dynamically increases the loads of the service based on the specified parameters, such as minQPS, maxQPS, adjustInterval, and qpsGrowthDelta, until the performance upper limit specified by maxRT or max QPS is exceeded or the request error rate specified by faultTolerate is exceeded. Sample configuration:
```
{
    "service": {
        "serviceName": "demo"
    },
    "data": {
        "content": "aGVsbG8K"
    },
    "optional": {
        "mode": "scan",
        "maxQPS": 1000,
        "minQPS": 500,
        "qpsGrowthDelta": 100,
        "adjustInterval": 30
    }
}
```
manual mode
The manual stress testing mode. The number of agent workers and the concurrency are fixed. You can dynamically adjust these values during the stress testing process. Sample configuration:
```
{
    "service": {
        "serviceName": "demo"
    },
    "data": {
        "content": "aGVsbG8K"
    },
    "optional": {
        "mode": "manual",
        "agentCount": 1,
        "concurrency": 5
    }
}
```