Create and manage stress testing tasks - Platform For AI

eas-benmark is a versatile tool for distributed stress testing that is embedded in Elastic Algorithm Service (EAS). With eas-benchmark, you can run dynamic stress tests, view real-time monitoring data that is accurate to the second, and generate a visualized report after the tests are completed. This helps you assess the performance of the service that you deploy in EAS. This topic describes how to create and manage stress testing tasks.

Prerequisites

A model service is deployed. For more information, see Model service deployment by using the PAI console.

Background information

You can log on to the EASCMD client and run the bench commands to create and manage stress testing tasks. For information about how to create and manage stress testing tasks, see the "Create a stress testing task" and "Manage stress testing tasks" sections of this topic. For information about how to log on to the EASCMD client, see Download the EASCMD client and complete identity authentication.

Create a stress testing task

Use the EASCMD client

Run the bench create command to create a stress testing task. After the task is created, you can use the returned URL to view the real-time monitoring data. In this example, the 64-bit Windows version of the EASCMD client is used.

eascmdwin64.exe bench create [bench_desc_json]

bench_desc_json specifies the name of the JSON configuration file for the stress testing task. Sample configuration:

{
    "service": {
        "serviceName": "xgb_test"
    },
    "data": {
        "content": "W1sxLDAsMCwwLDEsMSwwLDEsMCwxLDEsMCwwLDEsMCwxLDAsMSwwLDAsMSwxLDEsMCwxLDEsMCwwLDAsMSwxLDEsMCwxLDEsMSwxLDAsMSwxLDEsMCwxLDAsMCwwLDEsMSwwLDAsMCwxLDAsMSwwLDEsMCwwLDEsMCwwLDEsMCwxLDAsMCwxLDAsMCwwLDAsMSwwLDEsMCwxLDAsMCwxLDEsMSwwLDAsMSwwLDAsMCwwLDEsMSwxLDAsMSwxLDAsMCwxLDAsMSwwLDEsMSwxLDEsMCwxLDAsMCwxLDEsMSwxLDAsMCwwLDEsMSwwXV0K"
    }
}

You can use the path parameter to specify the path of multiple Object Storage Service (OSS) objects that store stress testing data, as shown in the following configuration. You can also set the path parameter to the path of the ZIP file that contains the stress testing data in the OSS://XX.zip format.

{
    "service": {
        "serviceName": "xgb_test"
    },
    "data": {
        "path": "oss://examplebucket/test1.bin,oss://examplebucket/test2.bin"
    }
}

The following table describes the parameters in the JSON configuration file.

Parameter	Nested parameter	Required	Description
service	serviceName	Yes	The name of the service for which you want to create the stress testing task.
data	content	No	A single entry of stress testing data in the Base64-encoded string format. If you want to configure multiple data entries, specify the path parameter. For information about the format and file type of stress testing data, see the "Stress testing data" section of this topic.
	path	No	The path of the stress testing data. You can specify an HTTP or OSS path. You can specify multiple paths. Separate the paths with commas (,). You can also package multiple stress testing files into a ZIP file and then specify the path of the ZIP file. Note Base64 encoding is not required for the stress testing data that is stored in files.
	multiLine	No	Specifies whether to separate the stress testing data by row. Valid values: true and false. Default value: false.
http	headers	No	The HTTP request headers. The value of this parameter is of the LIST type. Example: `["Authorization:aaa", "Content-Type:text"]`.
http	timeout	No	The HTTP request latency. Unit: milliseconds. Default value: 20000.
optional	mode	No	The stress testing mode. Valid values: auto: automatic mode. This is the default value. scan: periodic mode. manual: manual mode. For more information about the stress testing modes, see the "Stress testing modes" section of this topic.
	duration	No	The stress testing duration. Unit: seconds. Default value: 600. Maximum value: 1200.
	agentCount	No	If you set the mode parameter to manual, this parameter is required. The number of stress testing agents. Default value: 1. A higher value specifies a higher workload.
	concurrency	No	If you set the mode parameter to manual, this parameter is required. The concurrency of each stress testing agent. Default value: 2. A higher value specifies a higher workload. To scale the workload during stress testing, increase the concurrency of each stress testing agent before you increase the number of stress testing agents.
	adjustInterval	No	If you set the mode parameter to scan, this parameter is required. The interval at which the workload are automatically increased. Unit: seconds. Default value: 60.
	minQPS	No	If you set the mode parameter to scan, this parameter is required. The initial queries per second (QPS) for stress testing. Default value: 100.
	maxQPS	No	If you set the mode parameter to scan or auto, this parameter is required. The maximum QPS for stress testing.
	maxRT	No	If you set the mode parameter to scan or auto, this parameter is required. The maximum response time (TP99). If the actual response time exceeds this threshold, the QPS is automatically adjusted until the response time drops below the threshold.
	qpsGrowthDelta	No	If you set the mode parameter to scan, this parameter is required. The QPS increment. Default value: 50.
	faultTolerate	No	If you set the mode parameter to scan or auto, this parameter is required. The maximum tolerance rate for request errors. A request error occurs if the returned status code is not 200. A value of 0.01 specifies that the error handling mechanism is triggered when over 1% of requests encounter an error. Default value: 0.001.
	faultAction	No	If you set the mode parameter to scan or auto, this parameter is required. The operation of eas-benchmark when the request error rate exceeds the threshold that is specified by the faultTolerate parameter. Valid values: stop: maintains the current QPS. revise: dynamically adjusts the QPS until the request error rate drops below the threshold. This is the default value.

For example, if the JSON configuration file is named bench_xgb.json, you can run the following command to create a stress testing task:

eascmdwin64.exe bench create bench_xgb.json

Sample command output:

[RequestId]: DE240637-4976-59AF-A28C-BAA55C0A****
[OK] Task [benchmark-xgb-test-b514] is creating
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 1/1]: Benchmark task is Running
[OK] Benchmark task is Running
[OK] Click the link http://127.0.0.1:18222/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C.
Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command:
eascmd -c [config_file] bench visualize benchmark-xgb-test-b514

Use the PAI console

You can create a stress testing task that runs in automatic mode in the Platform for AI (PAI) console. To create a stress testing task in the PAI console, perform the following steps:

Go to the EAS-Online Model Services page.
1. Log on to the PAI console.
2. In the left-side navigation pane, click Workspaces. On the Workspaces page, click the name of the workspace that you want to manage.
3. In the left-side navigation pane, choose Model Deployment > Elastic Algorithm Service (EAS). The EAS-Online Model Services page appears.
Click the Stress Testing Task tab and then click Create Stress Testing Task.

In the Create Stress Testing Task panel, configure the following parameters and then click Create.

Parameter	Description
Service	The name of the service for which you want to create the stress testing task.
Testing Data	The stress testing data. Valid values: Single Data Entry, Data Address, OSS Object, and Upload Local File. For information about the format and file type of stress testing data, see the "Stress testing data" section of this topic. Note If you select Data Address or OSS Object, you can specify multiple file paths. Separate the paths with commas (,).
Specifies whether the data in the file needs to be separated by line breaks	This parameter is valid only if you set the Testing Data parameter to Data Address, OSS Object, or Upload Local File. If you turn on the switch, the data in the uploaded file is separated by rows and each row is used as a data entry for stress testing. If you turn off the switch, the entire content in the uploaded file is used as a data entry for stress testing.
Duration (s)	The stress testing duration. Default value: 300. Unit: seconds.
Maximum QPS	The maximum QPS for stress testing. Default value: 10000.
Maximum Response Time (ms)	The maximum response time. Unit: milliseconds. If the actual response time exceeds the value of this parameter, the QPS is automatically adjusted until the response time falls in the normal range.

Manage stress testing tasks

Use the EASCMD client

View the list of stress testing tasks

Run the bench list command to view the list of stress testing tasks that you created. In this example, the 64-bit Windows version of the EASCMD client is used.

eascmdwin64.exe bench ls

Sample command:

eascmdwin64.exe bench ls

Sample command output:

[RequestId]: 7F953F8E-8897-5785-808A-CA64830215F6
+-------------------------+--------------------------+-------------+----------------+---------+---------------------+
|        TASKNAME         |          TASKID          |   REGION    | AVAILABLEAGENT | STATUS  |     CREATETIME      |
+-------------------------+--------------------------+-------------+----------------+---------+---------------------+
| benchmark-xgb-test-7846 | eas-b-ql470xog6qeh25**** | cn-shanghai |              0 | Stopped | 2022-06-17 17:58:01 |
| benchmark-xgb-test-b514 | eas-b-bdnzvwq0z0h3xq**** | cn-shanghai |              2 | Running | 2022-06-20 12:18:54 |
+-------------------------+--------------------------+-------------+----------------+---------+---------------------+

View the details of a stress testing task

Run the bench desc command to view the details of a stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.

eascmdwin64.exe bench desc [benchmark_task_name]

benchmark_task_name specifies the name of the stress testing task whose details you want to view. Sample command:

eascmdwin64.exe bench desc benchmark-demo-test-c7eb

Sample command output:

+----------------+------------------------------------------------------------------------------+
|     TaskName   | benchmark-xgb-test-b514                                                      |
|     TaskId     | eas-b-bdnzvwq0z0h3xq****                                                     |
|    ServiceName | xgb_test                                                                     |
|         Region | cn-shanghai                                                                  |
|   DesiredAgent | 2                                                                            |
| AvailableAgent | 2                                                                            |
|         Status | Running                                                                      |
|        Message | Benchmark task is running                                                    |
|     CreateTime | 2021-10-20 12:38:35                                                          |
|     UpdateTime | 2021-10-20 12:38:45                                                          |
|         Config | {                                                                            |
|                |   "base": {                                                                  |
|                |     "agentCount": 2,                                                         |
|                |     "concurrency": 40,                                                       |
|                |     "duration": 1200,                                                        |
|                |     "requestCount":                                                          |
|                | 922337203685477****,                                                         |
|                |   },                                                                         |
|                |  ...                                                                         |
|                | }                                                                            |
+----------------+------------------------------------------------------------------------------+

Enable real-time visualization for a stress testing task
Run the bench visualize command to enable real-time visualization for a stress testing task. This command launches a web server on your local machine at 127.0.0.1, which provides a web page that displays the real-time monitoring data of the stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.
```
eascmdwin64.exe bench visualize [benchmark_task_name]
```
Sample command:
```
eascmdwin64.exe bench visualize benchmark-xgb-test-b514
```
Sample command output:
```
[OK] Click the link http://127.0.0.1:18734/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C.
Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command:
eascmd -c [config_file] bench visualize benchmark-xgb-test-b514
```
To view the real-time monitoring data, visit http://127.0.0.1:18734/eas-benchmark/statsview in a browser.
Obtain the stress testing report
When the status of a stress testing task changes to Stopped, the task is completed and the stress testing report is saved to OSS. You can run the bench report command to obtain the report. In this example, the 64-bit Windows version of the EASCMD client is used.
```
eascmdwin64.exe  bench report [benchmark_task_name]
```
Sample command:
```
eascmdwin64.exe bench report benchmark-demo-test-c7eb
```
Sample command output:
```
[OK] Benchmark task benchmark-demo-test-c7eb report url: http://eas-benchmark.oss-cn-chengdu.aliyuncs.com/summary/benchmark-demo-test-c7eb-10004.html
```
Copy the link after url: in the command output and enter the link in a browser to view the stress testing report, as shown in the following figure.

Adjust the stress testing workload

If you use the manual mode for stress testing, you can run the bench update command to modify the number of agents and the concurrency of each agent. In this example, the 64-bit Windows version of the EASCMD client is used.

eascmdwin64.exe bench update [benchmark_task_name] -Doptional.concurrency=<attr_value> -Doptional.agentCount=<attr_value>

<attr_value> specifies the value that you want to set. Sample command:

eascmdwin64.exe bench update benchmark-demo-b99c -Doptional.concurrency=2 -Doptional.agentCount=1

Sample command output:

[RequestId]: 9920C672-4D41-5CC4-8EC0-C690F76EB2BA
[OK] Running [TaskName: benchmark-demo-b99c, DesiredAgent:1, AvailableAgent: 1, Message: Benchmark task is Updating]
[OK] Benchmark task benchmark-demo-b99c was updated successfully

Stop a stress testing task
Run the bench stop command to stop a stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.
```
eascmdwin64.exe bench stop [benchmark_task_name]
```
Sample command:
```
eascmdwin64.exe bench stop benchmark-xgb-test-b514
```
Sample command output:
```
Are you sure to stop the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n]
[OK] Task [benchmark-xgb-test-b514] is stopping
[OK] [Agnet: 0/1]: Benchmark task is Running
[OK] [Agnet: 0/1]: Benchmark task is Stopped
[OK] Benchmark task is stopped
```
If you ran the command that enables real-time visualization in a terminal window before you stop the task, the system outputs the stress testing report in the terminal window. You can also run the bench report command to obtain the report.

Start a stress testing task

Run the bench start command to start a stopped stress testing task. In this example, the 64-bit Windows version of the EASCMD client is used.

eascmdwin64.exe bench start [benchmark_task_name]

Note

Compared with the bench create command, this command restarts a stress testing task based on the most recently updated configuration of the task.

Sample command:

eascmdwin64.exe bench start benchmark-xgb-test-b514

Sample command output:

Are you sure to start the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n]
[OK] Task [benchmark-xgb-test-b514] is starting
[OK] [Agnet: 0/1]: Succeed to start benchmark master
[OK] [Agnet: 1/1]: Benchmark task is Running
[OK] Benchmark task is Running
[OK] Click the link http://127.0.0.1:18947/eas-benchmark/statsview to observe realtime visualization details, you can turn it off with CTRL+C.
Turning off will not interrupt the benchmark test task, and you can reopen it by the visualize command:
eascmd -c [config_file] bench visualize benchmark-xgb-test-b514

Delete a stress testing task
After a stress testing task is completed, eas-benchmark retains records of the task based on the task status. The following table describes the retention periods.
Task status
Retention period
Stopped
48 hours
CreateFailed, UpdateFailed, Terminated, or Error
10 minutes
When the retention period ends, the system automatically deletes the task.
You can also run the bench delete command to manually delete a stress testing task. Format:
```
eascmdwin64.exe bench delete [benchmark_task_name]
```
Sample command:
```
eascmdwin64.exe bench delete benchmark-xgb-test-b514
```
Sample command output:
```
Are you sure to delete the benchmark task [benchmark-xgb-test-b514] in [cn-shanghai]? [Y/n]
[OK] Benchmark task benchmark-xgb-test-b514 is Deleting
[OK] Benchmark task was deleted successfully
```

Use the PAI console

On the Stress Testing Task tab, you can view and manage stress testing tasks.

You can start, stop, or delete a stress testing task with one click.
You can click the refresh button in the upper-right corner to refresh the status of stress testing tasks.
When the Status column of a task displays Running, you can click View in the Actions column to view the real-time monitoring data of the task.
When the Status column of a task displays Stopped, you can click View in the Actions column to view the stress testing report.

Stress testing data

Format

The format of stress testing data varies based on the model and the processor of your service.

If your service processes unstructured request data, such as audio, image, or text data, upload the files that contain the request data.
If your service processes structured request data, such as data in the TFRequest format, use EAS SDK to create the request data. You can also upload the generated binary file as the request data. For more information, see Warm up model services (advanced).

File type

Choose a file type for stress testing data based on your business requirements. Common file types include TEXT, JPG, and BIN.

Stress testing modes

When you use the EASCMD client to create a stress testing task, you can configure the stress testing mode by specifying the mode parameter that is nested in the optional parameter in the JSON configuration file. Valid values:

auto
Automatic mode. In this mode, eas-benchmark automatically creates the stress testing agents, configures the concurrency of each agent, and selects the optimal algorithm to determine the maximum capacity of the service. You need to only specify the service and data parameters in the JSON configuration file. Retain the default settings of other parameters. Sample configuration:
```
{
    "service": {
        "serviceName": "demo"
    },
    "data": {
        "path": "https://examplebucket.oss-cn-chengdu.aliyuncs.com/data/warmup.tf.bin"
    },
    "optional" {
        "maxQPS": 1000,
        "duration": 300
     }
}
```
scan
Periodic mode. In this mode, eas-benchmark dynamically increases the workload of the service based on parameters such as minQPS, maxQPS, adjustInterval, and qpsGrowthDelta, until the actual value of the maxRT, maxQPS, or faultTolerate parameter exceeds the specified one. Sample configuration:
```
{
    "service": {
        "serviceName": "demo"
    },
    "data": {
        "content": "aGVsbG8K"
    },
    "optional": {
        "mode": "scan",
        "maxQPS": 1000,
        "minQPS": 500,
        "qpsGrowthDelta": 100,
        "adjustInterval": 30
    }
}
```

manual

Manual mode. In this mode, you can manually adjust the number of agents and the concurrency of each agent during stress testing. Sample configuration:

{
    "service": {
        "serviceName": "demo"
    },
    "data": {
        "content": "aGVsbG8K"
    },
    "optional": {
        "mode": "manual",
        "agentCount": 1,
        "concurrency": 5
    }
}

References

After the performance of the service meets your business requirements, you can call the service to run inference tasks. For more information, see Overview.

Task status	Retention period
Stopped	48 hours
CreateFailed, UpdateFailed, Terminated, or Error	10 minutes