While a Java application is running, the Java virtual machine (JVM) process may consume a large amount of CPU. In this case, the performance of the Java application deteriorates. You can use some tools or commands to analyze and diagnose the issue, find the causes for the high CPU utilization of the JVM process, and resolve the issue.
Use the built-in commands of EDAS Agent for diagnosis
For Elastic Compute Service (ECS) instances in an ECS cluster, Enterprise Distributed Application Service (EDAS) displays the threads that consume CPU resources in JVM processes and the stack trace information of the threads. This allows you to quickly discover the causes for the high CPU utilization of JVM processes.
Use SSH to log on to the ECS instance where the process that consumes a large amount of CPU resides and run the su - admin command to switch to the admin account. Then, run the following command to view the top threads that consume the most CPU. Top five threads are displayed by default.
edas busy-threads
The following sample code provides an example of the system output:
09/28/19 22:57:07 [INFO] EXECUTING: busy-threads
[1] Busy(4.6%) thread(3222/0xc96) stack of java process(3221) under user(admin):
"main" #1 prio=5 os_prio=0 tid=0x00002ab68004e800 nid=0xc96 runnable [0x00002ab67c1df000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
at java.net.ServerSocket.implAccept(ServerSocket.java:545)
at java.net.ServerSocket.accept(ServerSocket.java:513)
at org.apache.catalina.core.StandardServer.await(StandardServer.java:490)
at org.apache.catalina.startup.Catalina.await(Catalina.java:819)
at org.apache.catalina.startup.Catalina.start(Catalina.java:765)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:309)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:443)
[2] Busy(0.9%) thread(2725/0xaa5) stack of java process(2721) under user(admin):
"DestroyJavaVM" #13 prio=5 os_prio=0 tid=0x00002ba81004c000 nid=0xaa5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
[3] Busy(0.0%) thread(3221/0xc95) stack of java process(3221) under user(admin):
[4] Busy(0.0%) thread(2721/0xaa1) stack of java process(2721) under user(admin):
The following sample code provides an example on how to use the commonly used parameters and options:
# Run the command once every two seconds for five times. By default, the top five threads that consume the most CPU in the JVM process are displayed each time.
edas busy-threads 2 5
# Display the top five threads that consume the most CPU in the specified JVM process. [$JVM_PID] indicates the ID of the JVM process. You can run ps -ef |grep java to view the ID of the JVM process.
edas busy-threads -p [$JVM_PID]
# Display the top 10 threads that consume the most CPU in a JVM process.
edas busy-threads -c 10
# Display the threads that consume a large amount of CPU within the most recent 2 seconds.
edas busy-threads --current
Use open source tools for diagnosis
show-busy-java-threads
In a non-ECS cluster, you can use the show-busy-java-threads open source script to discover the threads that consume the most CPU in a specified process. Run the following commands to download and use the show-busy-java-threads script. For more information about how to use the script, see show-busy-java-threads.
wget --no-check-certificate https://raw.github.com/oldratlee/useful-scripts/release/show-busy-java-threads
chmod +x show-busy-java-threads
./show-busy-java-threads
Arthas
In addition to the preceding script, you can use the Java diagnosis tool Arthas provided by Alibaba. This tool can also display the threads that consume the most CPU in a specified JVM process. Run the following command to display the top three threads that consume the most CPU and their stack trace information in the currently connected JVM process.
thread -n 3
The following sample code provides an example of the system output:
"as-command-execute-daemon" Id=29 cpuUsage=75% RUNNABLE
at sun.management.ThreadImpl.dumpThreads0(Native Method)
at sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:440)
at com.taobao.arthas.core.command.monitor200.ThreadCommand$1.action(ThreadCommand.java:58)
at com.taobao.arthas.core.command.handler.AbstractCommandHandler.execute(AbstractCommandHandler.java:238)
at com.taobao.arthas.core.command.handler.DefaultCommandHandler.handleCommand(DefaultCommandHandler.java:67)
at com.taobao.arthas.core.server.ArthasServer$4.run(ArthasServer.java:276)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Number of locked synchronizers = 1
- java.util.concurrent.ThreadPoolExecutor$Worker@6cd0b6f8
"as-session-expire-daemon" Id=25 cpuUsage=24% TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at com.taobao.arthas.core.server.DefaultSessionManager$2.run(DefaultSessionManager.java:85)
"Reference Handler" Id=2 cpuUsage=0% WAITING on java.lang.ref.Reference$Lock@69ba0f27
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@69ba0f27
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
Use the conventional diagnosis method
If your host is not connected to the Internet, you can use the following conventional method to obtain information about the threads that consume a large amount of CPU. Run the following command to collect information about the specified JVM process:
top -Hbp [$JVM_PID] -d 1 -n 1 >> top.[$JVM_PID].txt && jstack [$JVM_PID] >> jstack.[$JVM_PID].txt
- In the collected top.XXX.txt file, find the IDs of the threads that consume the most CPU. The threads in top.XXX.txt correspond to the stacks in jstack.XXX.txt.
- Run the following command to convert the decimal IDs of the threads to hexadecimal numbers.
printf %x [$Thread_ID]
Note [$Thread_ID] represents the ID of a thread that you found in Step 1. - Search for the hexadecimal IDs of the threads in jstack.XXX.txt and obtain information about the threads.