While a Java application is running, the Java virtual machine (JVM) process may consume a large amount of CPU. In this case, the performance of the Java application deteriorates. You can use some tools or commands to analyze and diagnose the issue, find the causes for the high CPU utilization of the JVM process, and resolve the issue.

Important This topic may contain information about third-party tools. The information is for reference only. Alibaba Cloud does not make guarantees or other forms of commitments for the performance and reliability of the third-party tools, and potential impacts of operations on these tools.

Use the built-in commands of EDAS Agent for diagnosis

For Elastic Compute Service (ECS) instances in an ECS cluster, Enterprise Distributed Application Service (EDAS) displays the threads that consume CPU resources in JVM processes and the stack trace information of the threads. This allows you to quickly discover the causes for the high CPU utilization of JVM processes.

Use SSH to log on to the ECS instance where the process that consumes a large amount of CPU resides and run the su - admin command to switch to the admin account. Then, run the following command to view the top threads that consume the most CPU. Top five threads are displayed by default.

edas busy-threads
Note Run the edas busy-threads -h command to view the help for using the command.

The following sample code provides an example of the system output:

09/28/19 22:57:07 [INFO]  EXECUTING: busy-threads
[1] Busy(4.6%) thread(3222/0xc96) stack of java process(3221) under user(admin):
"main" #1 prio=5 os_prio=0 tid=0x00002ab68004e800 nid=0xc96 runnable [0x00002ab67c1df000]
   java.lang.Thread.State: RUNNABLE
    at java.net.PlainSocketImpl.socketAccept(Native Method)
    at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
    at java.net.ServerSocket.implAccept(ServerSocket.java:545)
    at java.net.ServerSocket.accept(ServerSocket.java:513)
    at org.apache.catalina.core.StandardServer.await(StandardServer.java:490)
    at org.apache.catalina.startup.Catalina.await(Catalina.java:819)
    at org.apache.catalina.startup.Catalina.start(Catalina.java:765)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:309)
    at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)
[2] Busy(0.9%) thread(2725/0xaa5) stack of java process(2721) under user(admin):
"DestroyJavaVM" #13 prio=5 os_prio=0 tid=0x00002ba81004c000 nid=0xaa5 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
[3] Busy(0.0%) thread(3221/0xc95) stack of java process(3221) under user(admin):
[4] Busy(0.0%) thread(2721/0xaa1) stack of java process(2721) under user(admin):

The following sample code provides an example on how to use the commonly used parameters and options:

# Run the command once every two seconds for five times. By default, the top five threads that consume the most CPU in the JVM process are displayed each time. 
edas busy-threads 2 5

# Display the top five threads that consume the most CPU in the specified JVM process. [$JVM_PID] indicates the ID of the JVM process. You can run ps -ef |grep java to view the ID of the JVM process.
edas busy-threads -p [$JVM_PID]

# Display the top 10 threads that consume the most CPU in a JVM process. 
edas busy-threads -c 10

# Display the threads that consume a large amount of CPU within the most recent 2 seconds. 
edas busy-threads --current

Use open source tools for diagnosis

show-busy-java-threads

In a non-ECS cluster, you can use the show-busy-java-threads open source script to discover the threads that consume the most CPU in a specified process. Run the following commands to download and use the show-busy-java-threads script. For more information about how to use the script, see show-busy-java-threads.

wget --no-check-certificate https://raw.github.com/oldratlee/useful-scripts/release/show-busy-java-threads
chmod +x show-busy-java-threads
./show-busy-java-threads

Arthas

In addition to the preceding script, you can use the Java diagnosis tool Arthas provided by Alibaba. This tool can also display the threads that consume the most CPU in a specified JVM process. Run the following command to display the top three threads that consume the most CPU and their stack trace information in the currently connected JVM process.

Note Visit the Arthas official website to download Arthas.
thread -n 3

The following sample code provides an example of the system output:

"as-command-execute-daemon" Id=29 cpuUsage=75% RUNNABLE
    at sun.management.ThreadImpl.dumpThreads0(Native Method)
    at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:440)
    at com.taobao.arthas.core.command.monitor200.ThreadCommand$1.action(ThreadCommand.java:58)
    at com.taobao.arthas.core.command.handler.AbstractCommandHandler.execute(AbstractCommandHandler.java:238)
    at com.taobao.arthas.core.command.handler.DefaultCommandHandler.handleCommand(DefaultCommandHandler.java:67)
    at com.taobao.arthas.core.server.ArthasServer$4.run(ArthasServer.java:276)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
    Number of locked synchronizers = 1
    - java.util.concurrent.ThreadPoolExecutor$Worker@6cd0b6f8
"as-session-expire-daemon" Id=25 cpuUsage=24% TIMED_WAITING
    at java.lang.Thread.sleep(Native Method)
    at com.taobao.arthas.core.server.DefaultSessionManager$2.run(DefaultSessionManager.java:85)
"Reference Handler" Id=2 cpuUsage=0% WAITING on java.lang.ref.Reference$Lock@69ba0f27
    at java.lang.Object.wait(Native Method)
    -  waiting on java.lang.ref.Reference$Lock@69ba0f27
    at java.lang.Object.wait(Object.java:503)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) 

Use the conventional diagnosis method

If your host is not connected to the Internet, you can use the following conventional method to obtain information about the threads that consume a large amount of CPU. Run the following command to collect information about the specified JVM process:

top -Hbp [$JVM_PID] -d 1 -n 1 >> top.[$JVM_PID].txt && jstack [$JVM_PID] >> jstack.[$JVM_PID].txt
  1. In the collected top.XXX.txt file, find the IDs of the threads that consume the most CPU. The threads in top.XXX.txt correspond to the stacks in jstack.XXX.txt.
  2. Run the following command to convert the decimal IDs of the threads to hexadecimal numbers.
    printf %x [$Thread_ID]
    Note [$Thread_ID] represents the ID of a thread that you found in Step 1.
  3. Search for the hexadecimal IDs of the threads in jstack.XXX.txt and obtain information about the threads.