How to resolve errors in continuous profiling logs - Application Real-Time Monitoring Service

This topic answers common questions about using continuous profiling.

Enabling Continuous Profiling results in no data

Check that your configurations are correct. Ensure that the configured CIDR block includes the IP address of the application instance.
If you use an agent version earlier than 3.1.4, the profiling engine might fail because of a compatibility issue with the Alpine base image. This issue is fixed in version 3.1.4. To ensure stability and data integrity, we recommend that you upgrade the agent to version 3.1.4 or later.
Note
To check if you are using an Alpine Linux base image, see Other operations.
Continuous profiling uses an enhanced open source Async Profiler for data collection and does not support mounting multiple Async Profilers simultaneously. If your application also uses the continuous profiling feature from the Pyroscope agent, your application might fail to start.
On the continuous profiling page, set the query time 8 hours earlier and check if data appears. If data is displayed, the application's time zone might be set to UTC+0. This causes a data write delay of 8 hours compared to UTC+8.
Solution: Add an environment variable to the application to adjust the time zone to UTC+8.
Note
First, filter by the current pod name on the continuous profiling page to avoid confusion caused by multiple container restarts within the last 8 hours.
```
Key: JAVA_TOOL_OPTIONS, Value: -Duser.timezone=GMT+8
```
This issue is fixed in agent version 4.1.10. If you confirm this is the issue, you can also upgrade the agent to version 4.1.10 or later.
Check if the application has mounted other Async Profiler dynamic libraries:
1. Run the following command. Replace [pid] with the process ID of your application.
```
lsof -p [pid] | grep libasync
```
2. If the result contains a non-Alibaba Cloud dynamic library, such as the one shown below, your application is using its own Async Profiler dynamic library. This library is not compatible with Application Real-Time Monitoring Service (ARMS). To use ARMS features, you must remove your dynamic library.
```
/home/admin/xxx/.default/temp/libasyncProfiler1309163652530490111.so
```

Why have storage fees changed for the new continuous performance profiling?

The following are the main reasons for the change in the reported volume:

Continuous performance profiling is a new feature that is fully compatible with the original continuous profiling feature. It supports aggregate queries across multiple instances, thread-level analysis, and intelligent flame graph analysis with Copilot.
The storage structure has changed. To allow users to perform secondary analysis of profiling data, the storage medium is migrated from the built-in Object Storage Service (OSS) to a Simple Log Service (SLS) instance in your account. The SLS Project is `proj-xtrace-<encode>-<region-id>`, and the SLS Logstore is `logstore-profiling`. This change affects the volume of stored profiling data and the corresponding storage fees.

Supported time range for data queries

Continuous performance profiling data is stored for 7 days. You can query data from this period.

Is it normal for the memory usage from heap memory monitoring to differ from the memory usage detected by memory hot spots in continuous profiling?

Continuous Profiling records only heap memory allocations within a specified time period, not the process's total memory usage.

CPU diagnostics have data but memory diagnostics have no data

This issue usually occurs with agent versions 3.1.4 and later.

A lack of data in memory diagnostics is usually because you are using an Alpine base image. Alpine base images remove JDK debug symbols to reduce their size, which prevents continuous profiling from working correctly. To resolve this issue, you can install the JDK debug symbols in your image or use a non-Alpine base image. Note that some JDK versions do not have corresponding debug symbol packages, which can prevent a successful installation.

Note

To verify if the JDK in your environment contains debug symbols, see Check if the JDK in your environment includes debug symbols.

Why is there no data for code hot spots, or why does the data not meet expectations?

Hot spot profiling is not supported for applications that use virtual threads or similar technologies in the JDK, including the Alibaba Dragonwell JDK.
The code hot spot feature is not supported by the SkyWalking protocol. You can confirm the protocol type in the span details of a trace.
The code hot spot feature requires agent version 3.1.4 or later. Earlier versions do not support this feature.
Agent versions earlier than 4.2.1 support only synchronous calls and may not collect complete data. Data may be missing for asynchronous calls. For example, using Spring Cloud Gateway, Undertow, or Lettuce can cause asynchronous thread switching, which leads to inaccurate data collection. Agent versions 4.2.1 and later include optimizations for these issues. We recommend that you upgrade the agent to the latest version.
The code hot spot feature is supported only for traces that are sampled at a fixed rate. It is not supported for traces collected with non-fixed sampling rates, such as error sampling (s9) or slow sampling (s10), because this would cause high performance overhead. These sampling types are triggered only after a trace is complete. You can check the sample.reason field in the span's attributes to determine the sampling type.
For traces collected with a non-fixed sampling rate, go to the Trace Analysis page. Use the Traces with Code Hotspots parameter to filter for traces that contain code hot spots to diagnose issues.
If you use agent version 4.2.1 or later, the collected execution time might be much lower than the time recorded by the outer span. This can happen if your application uses an asynchronous, non-blocking I/O (NIO) framework, such as Spring Cloud Gateway. When these frameworks receive a request from an upstream service or send a request to a downstream service, they do not block the thread if data is not ready. Instead, the thread immediately returns and can perform other tasks, which improves thread utilization. Because the thread is not blocked waiting for I/O, the performance bottleneck is not in the current application. In this case, the time collected for the code hot spot can be much lower than the span duration. You should check for bottlenecks in the request processing of the upstream or downstream application, or in the network.

What is the performance overhead of continuous profiling?

Performance tests indicate the following overhead for continuous profiling: In a scenario with 500 Transactions Per Second (TPS) and all features enabled on a standard Spring Web application, the CPU overhead increases by approximately 5% and the off-heap memory overhead increases by approximately 50 MB. The increase in garbage collection (GC) and request latency is not significant.
In extreme cases, memory hot spot profiling lacks a rate limit. If an application allocates memory very frequently, it can generate a high volume of events, such as tens of thousands per minute. This can affect P99 latency. You can disable memory profiling to resolve this issue.
This issue is fixed in version 4.1.10. You can upgrade the agent to version 4.1.10 or later.

JFR-related threads in the application

The continuous profiling feature creates JDK Flight Recorder (JFR) threads. These threads are mainly found in agent versions before 4.1.10. Agent versions 4.1.10 and later do not create JFR threads.
These threads do not cause performance bottlenecks in the application.
If you dynamically turn off the continuous profiling switch, the thread is not destroyed immediately. It disappears only after the application restarts.

Continuous profiling for application startup time

Continuous profiling can slow down application startup. This can occur if the JDK lacks debug symbols and the Classloader loads many methods. After the application starts, there is no impact on its runtime performance.

This issue is fixed in agent version 4.2.1. You can upgrade the agent to this version to resolve the problem.

Why does the total memory in the flame graph exceed the actual configured memory limit?

The data in continuous profiling is not directly related to the host's memory configuration. It shows the total amount of heap memory allocated during the analysis period. Because of garbage collection, it is normal for this amount to be larger than the host's configured memory.

OpenJ9 JDK attachment failure

ARMS continuous profiling does not support IBM OpenJ9. Using this JDK can cause integration to fail and generate errors. We recommend that you use OpenJDK or Oracle JDK instead.

Code hot spot data in spans is incomplete

Code hot spots are identified by sampling the method stacks of threads in a trace at set intervals. If a method is missing from the code hot spot flame graph, it may be because its execution time is less than 500 ms. Methods with such short execution times may not be captured by the sampling process. This does not affect the diagnosis of slow traces because short-running methods are not considered performance bottlenecks.

Why does the flame graph contain .GC_active stacks?

.GC_active indicates that garbage collection (GC) was active during data collection for the flame graph. The GC process paused all Java business threads in an event known as "Stop-the-World". If .GC_active appears in a code hot spot, it means a GC pause contributed to the request's latency.

Why does the flame graph contain "unknown" entries?

On an Alpine operating system, if the JDK lacks debug symbols and the classloader loads many methods, enabling continuous profiling can slow down the application or cause it to time out. To prevent this, agent versions 4.2.1 and later monitor the loading time. The default threshold is 150 ms. If loading exceeds this time, the agent stops some time-consuming steps. This can prevent some method symbols from being parsed correctly, which results in "unknown" entries. You can adjust this threshold to mitigate this issue. For example, set the environment variable `AP_JMID_TIME_LIMIT` to 500 (milliseconds): `export AP_JMID_TIME_LIMIT=500`.

Why does the flame graph contain .no_Java_frame entries?

This usually happens when you use an Alpine base image. Alpine removes JDK debug symbols to reduce the image size. This prevents the system from identifying function names in the method stacks of C++ threads within the JDK. These stacks are then displayed as .no_Java_frame. These stacks mostly represent non-Java thread information, such as from the VM Thread or the Just-In-Time (JIT) compiler thread. If the proportion of .no_Java_frame entries is low, you can ignore them. Focus on the other Java method stacks for performance analysis. If the proportion is high, install debug symbols for the JDK in your base image. Or, switch to a non-Alpine base image. Note that some JDK versions do not have corresponding debug symbol packages, which can prevent installation.

Why does the "other" item appear in the flame graph?

Problem

An "other" item appears in the flame graph. For example, in the analysis results with Allocated Memory selected as the profiling type, the other item (6.41 MB) is highlighted in orange in the memory allocation ranking table, placed between specific Java methods (such as AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter() 8.99 MB and java.util.Arrays.copyOfRange(char[], int, int) 8.90 MB). The flame graph on the right displays the corresponding call stack hierarchy.

Cause

The "other" item in a flame graph is normal. A flame graph has a tree structure. When there are too many nodes, it can be difficult to identify key information. ARMS combines less significant nodes into the "other" category to simplify the graph and highlight important information.

Log output: parse lib sigsegv handler installed

The ARMS agent prints this informational log message. It appears only after you enable continuous profiling and does not affect the application's runtime performance. ARMS will disable this log output in a future version.

How do I resolve the "No access to perf events" error caused by perf_event_open restrictions?

Problem

Async-Profiler depends on the perf_event_open system call for CPU profiling. However, security policies in the Linux kernel, such as seccomp, can block processes from using certain system calls.

The error message is as follows:

[ERROR] Failed to execute 'start,jfr=0,event=cpu,interval=11ms,alloc=512k,file=/tmp/cpc-async-profiler-7729534006755968198.jfr'
[ERROR] Failed to start Continuous Profile Collector
java.lang.RuntimeException: java.lang.IllegalStateException: No access to perf events. Try --fdtransfer or --all-user option or 'sysctl kernel.perf_event_paranoid=1'

Solutions

For Docker environments, run the container with the following command. For more detailed control over system calls, see the official documentation.
```
  docker run --security-opt seccomp=unconfined  XXX
```
For Kubernetes environments, set the privileged container parameter to privileged: true. Privileged containers are always Unconfined.
For more detailed control over system calls, see the official documentation.

How do I resolve the "No AllocTracer symbols found. Are JDK debug symbols installed?" error?

This error, or a lack of data, can occur if your Java process runs in a container that uses an Alpine base image. Alpine base images remove JDK debug symbols to reduce their size, which prevents continuous profiling from working correctly. To resolve this issue, see the solutions for memory hot spot collection failure due to missing debug symbols.

How do I resolve the "perf_event mmap failed..." error?

Problem

This error usually appears in the standard output of the Java Virtual Machine (JVM). When continuous profiling samples CPU hot spots, it collects both native stacks (Linux Kernel, JVM, and C/C++) and Java stacks. To collect native stacks, it must perform a memory map (MMap) on the perf_event file descriptor for each Java thread. The Linux kernel limits the total memory for these MMap operations. The default limit is 516 KB. If your Java application has many threads, it can exceed this limit. This triggers the perf_event mmap failed... warning in the Java standard output. This warning does not affect your Java application or business logic. The only side effect is that native stacks will not appear in the flame graph. Because the Java method stack is usually sufficient for diagnosing CPU hot spots, you can typically ignore this warning.

Solutions

To eliminate this error message, perform the following steps:

On the host machine, run the following command.
```
echo 1028 > /proc/sys/kernel/perf_event_mlock_kb
```
The default threshold is 516. You can increase this value until the warning stops. The value should conform to the formula 8*N + 4, where N is a natural number. For example, 516 = 512 + 4, and 1028 = 1024 + 4.
To resolve the error, restart Docker.

Other operations

Check if the JDK in your environment includes debug symbols

Missing debug symbols will cause memory hot spot activation to fail. As a result, no memory hot spot data can be collected.

Check the agent logs for missing debug symbols.
In the agent installation directory, check the `logs` directory for a `cpc.log` file. In some older agent versions, this file is in the `logs/arms_log` directory. If the log contains the keyword No AllocTracer symbols found. Are JDK debug symbols installed?, it means debug symbols are missing.
```
2023-03-08 11:32:00 [continuous Profile collector] [INFO]
Some or all engines success, details below
CPU:true
Alloc:false:No AllocTracer symbols found. Are JDK debug symbols installed?
Wall:true
Other:false
Lock:false
```

Use commands to check if the environment is missing debug symbols.

Go to the JDK installation path.
You can find the JDK installation path in your environment using the which java command or echo $JAVA_HOME.
From the top level of the JDK installation path, run the following command to find the path to the `libjvm.so` file.
```
find ./ -name "*libjvm.so*"
```

Run the following command to check if debug symbols have been stripped from the file. Replace /path/to/ with the actual path.

file /path/to/libjvm.so

If the command returns not stripped, the JDK in your environment includes debug symbols. If it returns stripped, it does not include debug symbols.

[root@iZj6cbirlzrhmgljxdfxw9Z jdk-directory]# cd jdk8u412-b08
[root@iZj6cbirlzrhmgljxdfxw9Z jdk8u412-b08]# ls
ASSEMBLY_EXCEPTION  LICENSE  NOTICE  THIRD_PARTY_README  bin  include  jre  lib  man  release  sample  src.zip
[root@iZj6cbirlzrhmgljxdfxw9Z jdk8u412-b08]# find ./ -name "*libjvm.so*"
./jre/lib/amd64/server/libjvm.so
[root@iZj6cbirlzrhmgljxdfxw9Z jdk8u412-b08]# file ./jre/lib/amd64/server/libjvm.so
./jre/lib/amd64/server/libjvm.so: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, not stripped

Solutions for memory hot spot collection failure due to missing debug symbols

Method 1: Upgrade to JDK 11 or later

The implementation of JDK 11 and later has been adjusted and no longer depends on debug symbol information.

Method 2: Change the base image

You can go to Docker Hub and use the `openjdk` keyword to find popular JDK distributions, such as eclipse-temurin, ibm-semeru-runtimes, and amazoncorretto. Then, search for a JDK image that is not built on Alpine Linux. If you have your own JDK image library, you can also find a replacement there. JDK images built on Alpine Linux usually have `alpine` in their tags.

For example, the eclipse-temurin:8u382-b05-jdk-alpine image has alpine in its tag, which indicates that it is built on Alpine Linux. Avoid using this image.

Example of a JDK version built on a non-Alpine Linux base image:

For example, the eclipse-temurin:8u392-b08-jdk image on Docker Hub does not have alpine in its tag. It is a JDK version built on a non-Alpine Linux base image and supports multiple architectures, including linux/amd64, linux/arm/v7, linux/arm64/v8, linux/ppc64le, and windows/amd64.

If you still have no data after you change the base image, check the local cpc.log file for the No AllocTracer symbols found. Are JDK debug symbols installed? error message. This message indicates that debug symbols are still missing in your environment. In this case, use another base image or Method 3: Use a standard Alpine Linux and JDK.

Method 3: Use a standard Alpine Linux and JDK

Use ARMS agent version 3.2.8 or later and declare Alpine Linux and the JDK in the Dockerfile.

from Alpine:3.9
RUN apk add openjdk8