Apache JMeter is an open source stress testing tool that supports parameterization, assertions, and extensions for a variety of protocols and controllers, and it lets you write custom scripts to implement logic such as parameter processing. PTS runs your JMeter scripts on the native JMeter engine and adds horizontal scaling and Cloud Monitor integration for high-concurrency testing and monitoring diagnostics. This topic describes the use cases of native JMeter stress testing in PTS and how to create a JMeter scenario, start a stress test, and work with the stress testing report.
Use cases
Create a JMeter scenario in PTS in the following situations:
High concurrency — You need distributed stress testing at high concurrency.
Real-time observability — You need second-level real-time monitoring during the stress test, and a stress testing report that is generated automatically after the stress test.
Centralized management — You need to manage JMeter scripts and JMeter environment dependencies centrally.
Limits
Before you create a JMeter scenario, review the following limits:
JMX script files — The file size cannot exceed 2 MB. You can upload multiple JMX script files, but only one JMX script can be used for a stress test.
JAR files — The file size cannot exceed 10 MB. Before you upload a JAR file, debug it in your on-premises JMeter environment to make sure that the debugging is passed.
CSV files — The file size cannot exceed 60 MB. If a file exceeds 60 MB, use the OSS data source feature to import the file.
Number of files — The total number of non-JMX files cannot exceed 20.
Step 1: Create a JMeter scenario
On the Create JMeter Scenario page, enter a name in the Scenario field.
In the Scenario Settings section, upload the JMeter stress testing script. The file name cannot contain space characters, and the extension must be
jmx.After the JMX stress testing file is uploaded, PTS automatically completes the missing plug-ins for you.
If you upload multiple JMX files, select one JMX file as the JMeter script that is used to start the stress testing scenario.
Click Upload File to upload other files, such as data files in the CSV format and plug-ins in the JAR format.
ImportantAn uploaded file with the same name overwrites the previously uploaded file. To compare files, obtain the MD5 value in the Actions column of the file and compare it with the MD5 value of your local file to confirm whether the file is changed.
Note:
When you prepare a CSV file, do not directly change the extension of an XLSX file. Export the file by using software such as Excel or Numbers. If the file is generated by a program, you can use Apache Commons CSV.
If a data file is associated with the JMX script, you must change Filename in CSV Data Set Config to the file name itself and do not include the path. Otherwise, the data file cannot be read. Similarly, if you use the
__CSVReadfunction or specify files in a JAR file, you must use the file name.
(Optional) Select Split File for a CSV file to make sure that the data in the file is not duplicated on each load generator. If you do not select this option, the same data is used on each load generator.
Select whether to use a dependency environment.
If you select Yes, you must select an environment that is created in environment management.
If you select No, you must select a JMeter version. Apache JMeter 5.0 and Java 11 are supported.
Configure the load for the scenario.
(Optional) Add cloud resource monitoring. By adding monitoring, you can quickly view the corresponding monitoring data during the stress testing process and in the stress testing report. This feature is disabled by default. To configure it, turn on the switch.
(Optional) Add other information. You can add the stress testing owner and remarks here. This feature is disabled by default. To configure it, turn on the switch.
(Optional) Configure advanced settings. Advanced settings include log sampling rate settings, DNS configuration, and distributed adaptation component settings. Advanced settings are disabled by default. To configure them, turn on the switch.
The following tabs describe the default values that apply when advanced settings are disabled and the items that you can configure when advanced settings are enabled.
Advanced settings disabled
The default settings are used:
Sampling Log Sampling Rate is set to
1, and the fee is calculated as(1+rate)*native VUM.The Clear DNS Cache in Each Loop switch is turned off, and the system DNS resolver is used.
Both the Synchronous Timer and the Constant Throughput Timer take effect on a single generator.
Advanced settings enabled
Log sampling rate settings
You can customize the sampling rate of sampling logs. The default sampling rate is 1%. To reduce the sampling rate, enter a value in the
(0,1]range. To increase the sampling rate, enter a value in the(1,50]range that is divisible by 10, such as 20.If you specify a sampling rate greater than 1%, an additional fee of
rate%is charged. For example, if the sampling rate is 20%, an additional fee of20%*VUMis charged.DNS cache settings
You can select whether to clear the DNS cache for each access request. If you need to clear the DNS cache, you can use the system DNS resolver or a custom DNS resolver. You may need a custom DNS resolver in the following scenarios:
Distributed adaptation component settings
If the script that you upload contains distributed adaptation components, such as timers or controllers, you can configure distributed adaptation for multiple load generator IP addresses. This helps you run more accurate and effective performance tests. Select Takes effect globally if the value configured in the script is the threshold of the whole load generator cluster. Select Single Generator if the value configured in the script is the threshold of each load generator. By default, both timers take effect on a single generator.
Configure the Synchronous timer. If the JMeter script that you upload contains a timer, you must select whether the value configured in the script takes effect globally or takes effect on a Single Generator.
Takes effect globally: The value configured in the script is the overall threshold of the cluster. The value is split into single-generator thresholds based on the number of IP addresses that are used. That is, the threshold of a single load generator is the value in the script divided by the number of IP addresses. If two load generator IP addresses are used in total and the aggregate threshold configured in the script is 1,000, the threshold of a single load generator is 500.
Single Generator: The value configured in the script is the threshold of a single load generator, and the script content is not replaced. In this case, check whether the concurrency matches the configured value. If two load generator IP addresses are used in total and the threshold configured in the script is 1,000, the threshold of a single load generator is 1,000, and the threshold of the whole load generator cluster is 2,000.
Configure the Constant Throughput Timer. If the JMeter script that you upload contains a controller, you must select whether the value configured in the script takes effect globally or takes effect on a Single Generator.
Takes effect globally: The value configured in the script is the overall threshold of the cluster. The value is split into single-generator target throughput values based on the number of load generator IP addresses that are used. That is, the threshold of a single load generator is the value in the script divided by the number of IP addresses. If two load generator IP addresses are used, the concurrency is 100, the script contains only one thread group, its target throughput is 100 per minute, and the calculation mode is the current thread, the total target throughput of the scenario per minute is 2 × (100 concurrent users/2) × (100/2) = 5,000.
Single Generator: The value configured in the script is the target value of a single load generator, and the script content is not replaced. Check whether the concurrency level matches the configured value. If two load generator IP addresses are used, the concurrency is 100, the script contains only one thread group, its target throughput is 100 per minute, and the calculation mode is the current thread, the total target throughput of the scenario per minute is 2 × (100 concurrent users/2) × 100 = 10,000.
Step 2: Start the stress test
Click Debug to go to the scenario debugging page and verify whether the configuration is valid. Debugging a scenario checks issues such as network connectivity, plug-in integrity, and the correctness of the script configuration in advance, which prevents the stress test from failing to start. For instructions, see Debug a scenario.
Click Save and Start. On the Note page, select Execute Now and select The test is permitted and complies with the applicable laws and regulations., and then click Start Test.
Monitor a running stress test
During the stress testing process, you can monitor the stress testing data of the JMeter scenario and perform operations such as scenario speed regulation.
Data
The following table describes the data that PTS displays during the stress test.
Data information | Description |
Real-time VUM | The total number of resources consumed by this stress test, in VUM, which stands for virtual user minutes. |
Request success rate (%) | The request success rate of all scenarios within the statistical period of all load generators. |
Average RT (success/failure) | Success RT Avg (ms): the average RT of all successful requests. Failure RT Avg (ms): the average RT of all failed requests. |
Transactions Per Second (TPS) | The total number of requests within the statistical period of all load generators divided by time, in seconds (s). |
Anomalies | The number of abnormal requests. Common causes include connection timeouts and request timeouts. |
Traffic (Request/Response)/Seconds | The traffic of the requests sent and the traffic of the responses received by the load generator. |
Concurrency (current/maximum) | The concurrency value of the load. If scenario speed regulation is configured during the stress testing process, the current concurrency value and the configured maximum concurrency value are displayed separately. In the warm-up phase, the configured concurrency is not reached. After the warm-up ends, the concurrency is the configured value. To adjust the scenario concurrency during the stress testing process, click Speed Regulation and enter the required concurrency number. |
Total requests | The total number of requests sent by the entire scenario during the stress testing process. |
The monitoring data is based on simple aggregate calculation of the Backend Listener. The statistical sampling period of the load generator is 15s, and the period of data aggregation and calculation is also 15s. Therefore, data latency may occur.
Configuration Information
The configuration information page lists the basic information in the scenario configuration, including the load source, the configured duration, the traffic model, and the number of specified IP addresses.
Stress testing information
On the Testing Details tab, click View Chart on the right side of a single trace to view its real-time data, such as TPS, success rate, response time, and traffic.
Click the Load Generator Performance tab to view information such as the CPU utilization, Load5, memory utilization, and network traffic time series curves of all load generators during the stress testing process. You can also filter the performance information of a specific load generator.
If you added ECS monitoring, SLB monitoring, RDS monitoring, or ARMS monitoring, you can click the Cloud Resource Monitoring tab to view the related monitoring information.
Sampling log details
Select the Sampling Logs tab.
Filter logs by conditions such as the stress testing sampler, the response status, and the RT range.
Click Click to view details in the Actions column of the corresponding log.
In the Log Details dialog box, view the log fields and field values on the General tab. In the upper-right corner of the dialog box, switch the display format to General or the HTTP protocol style.
If Embedded Resources from HTML Files is configured in the JMeter script, the Sub-request Details tab appears in the log details. By selecting a specific subrequest, you can filter out the corresponding request logs.
The log details also provide the following tabs:
The Timing Waterfall tab displays the time consumption of the total request and each subrequest.
The Trace View tab displays the call chain details of the upstream and downstream of the endpoint under stress testing.
Step 3: View the stress testing report
After the stress test stops, PTS automatically collects the data generated during the stress testing process to form a stress testing report. The report presents the overall dynamic data of the JMeter stress testing scenario and the details of each request.
On the Reports page, select the JMeter scenario type. In the Actions column of the target report, click View Report.
In a trend chart, the statistical sampling period of each data point is 15s. Data latency may occur in the initial phase of the final data report.
(Optional) Step 4: Export the stress testing report
Stress testing report data is retained for 30 days. If you need to keep a report for a longer period, export the current stress testing report to your local machine.
On the Report Details page, click Export Report.
Select Version with Watermark or Version without Watermark to download the stress testing report as a PDF file to your local machine.