This topic provides the performance test report of the Application Real-Time Monitoring Service (ARMS) agent for Java. In this report, the testers performed multiple rounds of stress testing on the same application. The test results show the performance of the ARMS agent for Java when the application processes different amounts of transactions.

Background information

Testing environment

  • Version of the ARMS agent for Java: V2.7.3.
  • Testing platform: Alibaba Cloud Performance Testing (PTS)
  • Hardware configurations of the ECS instance that hosts the application: 2 Intel(R) Xeon(R) Platinum 8369B CPU @ 2.70 GHz, 4 GB memory
  • Operating system version: CentOS Linux release 7.6.1810 (Core)

Application

The application is developed based on the Spring framework, including Spring Boot, Spring MVC, a simulated Redis client, and a JDBC connection pool. The JDBC connection pool is used to simulate a MySQL client. When ARMS monitors the application, the ARMS agent for Java captures seven spans for each transaction, including one Tomcat, one Spring MVC, one Method, two Jedis, and two MySQL.

Download the application for testing

Baseline metrics

A baseline is a benchmark metric obtained by testing the application without installing the ARMS agent. The following metrics are used:
  • CPU: CPU utilization
  • MEM: memory in use
  • TPS: transactions per second
  • RT: response time, in milliseconds

Calculation method of performance overhead

To calculate the performance overhead of the ARMS agent for Java, the following steps are performed: obtain the baseline metrics, install the agent for the application, perform stress testing on the application, and then observe the CPU, MEM, TPS, and RT metrics.

Performance overhead of the agent = (Test result after the agent is installed - Baseline metric) / Baseline metric

Baseline metrics

The testers tested the performance of the application at 500, 1,000, and 2,000 TPS. The test results are used as baseline metrics for subsequent comparative experiments. The following figure shows the test results for baseline metrics.Baseline metrics

Test results with the ARMS agent for Java installed

The application is tested based on the following scenarios. In each scenario, tests are performed at 500, 1,000, and 2,000 TPS.
  • Scenario 1: Use the default configurations of the ARMS agent for Java. The sampling rate is 10%.
  • Scenario 2: Use the default configurations of the ARMS agent for Java. The sampling rate is 100%.
Note To view the default configurations of the ARMS agent for Java, log on to the ARMS console and navigate to the Application Settings > Custom Configurations page.

Test results of Scenario 1

500 The following figure compares the test results with the ARMS agent for Java installed and the test results for baseline metrics at 500 TPS.

Scenario 1 - 500 TPS

At 500 TPS, the extra CPU overhead of the ARMS agent for Java is 1.49% (6.94% of extra loss compared to the baseline metric), the extra memory overhead is 1.2%, and the RT loss is about 1 millisecond.

The following figure compares the test results with the ARMS agent for Java installed and the test results for baseline metrics at 1,000 TPS.

Scenario 1 - 1,000 TPS

At 1,000 TPS, the extra CPU overhead of the ARMS agent for Java is 2.12% (6.54% of extra loss compared to the baseline metric), the extra memory overhead is 1.18%, and the RT loss is about 1 millisecond.

The following figure compares the test results with the ARMS agent for Java installed and the test results for baseline metrics at 2,000 TPS.

Scenario 1 - 2000 TPS

At 2,000 TPS, the extra CPU overhead of the ARMS agent for Java is 3.67% (6.90% of extra loss compared to the baseline metric), the extra memory overhead is 1.2%, and the RT loss is 0 milliseconds.

Test results of Scenario 2

500 The following figure compares the test results with the ARMS agent for Java installed and the test results for baseline metrics at 500 TPS.

Scenario 2 - 500 TPS

At 500 TPS, the extra CPU overhead of the ARMS agent for Java is 1.51% (7.03% of extra loss compared to the baseline metric), the extra memory overhead is 1.25%, and the RT loss is 0 milliseconds.

The following figure compares the test results with the ARMS agent for Java installed and the test results for baseline metrics at 1,000 TPS.

Scenario 2 - 1000 TPS

At 1,000 TPS, the extra CPU overhead of the ARMS agent for Java is 2.85% (8.79% of extra loss compared to the baseline metric), the extra memory overhead is 1.54%, and the RT loss is about 1 millisecond.

The following figure compares the test results with the ARMS agent for Java installed and the test results for baseline metrics at 2,000 TPS.

Scenario 2 - 2000 TPS

At 2,000 TPS, the extra CPU overhead of the ARMS agent for Java is 4.64% (8.72% of extra loss compared to the baseline metric), the extra memory overhead is 1.15%, and the RT loss is 0 milliseconds.

Summary

CPU

Item 500 TPS 1,000 TPS 2,000 TPS
Baseline 21.47% 32.43% 53.21%
Scenario 1 22.96% 34.55% 56.88%
Scenario 2 22.98% 35.28% 57.85%

Memory (MEM)

Item 500 TPS 1,000 TPS 2,000 TPS
Baseline 53.51% 53.62% 54.09%
Scenario 1 54.71% 54.80% 55.29%
Scenario 2 54.76% 55.16% 55.24%

RT

Item 500 TPS 1,000 TPS 2,000 TPS
Baseline 107 ms 109 ms 129 ms
Scenario 1 108 ms 110 ms 129 ms
Scenario 2 107 ms 110 ms 129 ms

The application is tested at different TPS. The default configurations of the ARMS agent for Java are used for both scenarios whereas the sampling rate is different. The ARMS agent for Java is capable of providing most services with 5% to 10% of CPU overhead and less than 100 MB of memory overhead (the ARMS agent for Java occupies about 30 MB of metaspace).