Java threads

This article has been “at the bottom of my closet” for a long time. Recently some colleagues from our company's technical community have posted several articles sharing their ideas on Java threads and I found more can be discussed about this topic. So I spent some time to finish this article. This article only focuses on thread models on the JVM and does not mean to reflect the thread models in real operating systems (Due to the limited length of this article, the thread dump structures won’t be discussed, nor will the comprehensive usage of tools during the optimization process, such as ps, perf.top, iostat, jstack, TDA plugin, and thread inspector. If you have any questions, please feel free to post a reply for discussion). I may consider conducting in-depth analysis on the thread/process models on the xUnix and Windows platforms and hope you will like it.
Okay. Let's get down to the business. Start with the following picture:


Java threads have a total of six states, namely New, Runnable, Blocked, Waiting, Timed_Waiting and Terminated. We will focus on the Blocked and Timed_Waiting states.
Blocked state: There are generally two preconditions for the thread to enter this state: first, waiting for monitor (intrinsic or external); second, using entry or reenter to synchronize code blocks. Here let's first look at the two queues in the Java thread model, as shown in the figure below:


Each monitor can only be owned by one thread at a time, and the owner thread is the “active thread”, and all the other threads are “waiting threads” waiting in two queues, namely the “entry set” and the “wait set” respectively. The state of threads in the “entry set” is “Waiting for monitor entry”, while the state of threads waiting in the “wait set” is “in Object.wait()”. If you have used the ReentrantLock or ReentrantReadWriteLock classes improperly, your thread may be stuck in the Blocked state which is also a frequently-encountered case during our optimization. The solution is also very easy, just to locate the address of the thread to be locked and analyze whether thread starvation has occurred.
The official documents provide detailed explanations on the Timed_Waiting state, that is, the thread will enter the state when you call the methods below.
Thread.sleep
 Object.wait with timeout
 Thread.join with timeout
 LockSupport.parkNanos
 LockSupport.parkUntil

Here we should pay attention to the LockSupport class. The class is a basic thread congestion primitive used to create locks and other sync classes, and an optimization for Thread.suspend and Thread.resume(). It is also an optimization targeting busy wait and preventing excessive spin (If you are interested in this topic, you can refer to the Reference document No 3).
Okay. After briefing the several key thread states, we can now start to look at the thread stack through several cases:
Case 1: Acceptor in the NIO
"qtp589745448-36 Acceptor0 SelectChannelConnector@0.0.0.0:8161" prio=10 tid=0x00007f02f8eea800 nid=0x18ee runnable [0x00007f02e70b3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
- locked <0x00000000ec8ffde8> (a java.lang.Object)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
- None

The implementation in the source code is as follows:
public void accept(int acceptorID) throws IOException
100    {
101        ServerSocketChannel server;
102        synchronized(this)
103        {
104            server = _acceptChannel;
105        }
106
107        if (server!=null && server.isOpen() && _manager.isStarted())
108        {
109            SocketChannel channel = server.accept();
110            channel.configureBlocking(false);
111            Socket socket = channel.socket();
112            configure(socket);
113            _manager.register(channel);
114        }
115    }

I want to emphasize on the thread stack that the NID (native LWP ID) refers to native lightweight process (that is, thread) ID.
Case 2: Selector in the NIO
"qtp589745448-35 Selector0" prio=10 tid=0x00007f02f8ee9800 nid=0x18ed runnable [0x00007f02e71b4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000000ec9006f0> (a sun.nio.ch.Util$2)
- locked <0x00000000ec9006e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000000ec9004c0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
- None

The code snippets are as follows:
// If we should wait with a select
566                     if (wait>0)
567                     {
568                         long before=now;
569                         selector.select(wait);
570                         now = System.currentTimeMillis();
571                         _timeout.setNow(now);
572
573                         // If we are monitoring for busy selector
574                         // and this select did not wait more than 1ms
575                         if (__MONITOR_PERIOD>0 && now-before <=1)
576                         {
577                             // count this as a busy select and if there have been too many this monitor cycle
578                             if (++_busySelects>__MAX_SELECTS)
579                             {
580                                 // Start injecting pauses
581                                 _pausing=true;
582
583                                 // if this is the first pause
584                                 if (!_paused)
585                                 {
586                                     // Log and dump some status
587                                     _paused=true;
588                                     LOG.warn("Selector {} is too busy, pausing!",this);
589                                 }
590                             }
591                         }
592                     }

Case 3: Handlers for the MQTT protocol in ActiveMQ
"ActiveMQ Transport Server Thread Handler: mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600" daemon prio=10 tid=0x00007f02f8ba6000 nid=0x18dc waiting on condition [0x00007f02ec824000]
   java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x00000000faad0458> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:373)
at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
- None

Code snippets:
@Override
    protected void doStart() throws Exception {
        if (useQueueForAccept) {
            Runnable run = new Runnable() {
                @Override
                public void run() {
                    try {
                        while (!isStopped() && !isStopping()) {
                            Socket sock = socketQueue.poll(1, TimeUnit.SECONDS);
                            if (sock != null) {
                                handleSocket(sock);
                            }
                        }

                    } catch (InterruptedException e) {
                        LOG.info("socketQueue interuppted - stopping");
                        if (!isStopping()) {
                            onAcceptError(e);
                        }
                    }
                }
            };
            socketHandlerThread = new Thread(null, run, "ActiveMQ Transport Server Thread Handler: " + toString(), getStackSize());
            socketHandlerThread.setDaemon(true);
            socketHandlerThread.setPriority(ThreadPriorities.BROKER_MANAGEMENT - 1);
            socketHandlerThread.start();
        }
        super.doStart();
    }

Case 5: Simulate bank transfer and deposit.
"withdraw" prio=10 tid=0x00007f3428110800 nid=0x2b6b waiting for monitor entry [0x00007f34155bb000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at com.von.thread.research.DeadThread.depositMoney(DeadThread.java:13)
- waiting to lock <0x00000000d7fae540> (a java.lang.Object)
- locked <0x00000000d7fae530> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:28)
at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
- None

"deposit" prio=10 tid=0x00007f342810f000 nid=0x2b6a waiting for monitor entry [0x00007f34156bc000]
   java.lang.Thread.State: BLOCKED (on object monitor)
at com.von.thread.research.DeadThread.withdrawMoney(DeadThread.java:21)
- waiting to lock <0x00000000d7fae530> (a java.lang.Object)
- locked <0x00000000d7fae540> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:29)
at java.lang.Thread.run(Thread.java:724)

   Locked ownable synchronizers:
- None

Found one Java-level deadlock:
=============================
"withdraw":
  waiting to lock monitor 0x00007f3400003620 (object 0x00000000d7fae540, a java.lang.Object),
  which is held by "deposit"
"deposit":
  waiting to lock monitor 0x00007f3400004b20 (object 0x00000000d7fae530, a java.lang.Object),
  which is held by "withdraw"

Java stack information for the threads listed above:
===================================================
"withdraw":
at com.von.thread.research.DeadThread.depositMoney(DeadThread.java:13)
- waiting to lock <0x00000000d7fae540> (a java.lang.Object)
- locked <0x00000000d7fae530> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:28)
at java.lang.Thread.run(Thread.java:724)
"deposit":
at com.von.thread.research.DeadThread.withdrawMoney(DeadThread.java:21)
- waiting to lock <0x00000000d7fae530> (a java.lang.Object)
- locked <0x00000000d7fae540> (a java.lang.Object)
at com.von.thread.research.DeadThread.run(DeadThread.java:29)
at java.lang.Thread.run(Thread.java:724)

Found 1 deadlock.

Here it is a deadlock triggered by non-sequential locking.


       Well, it is about time to summarize the three cases deserving special attention during the optimization process (grep java.lang.Thread.State dump.bin | awk ‘{print $2$3$4$5}’ | sort | uniq -c):
       1. Waiting for monitor entry – thread is in the Blocked state. Possible issues: Deadlock (sequential deadlock, starvation deadlock...)
       2. Waiting on condition – sleeping or Timed_Waiting state. Possible issues: IO bottleneck
       3. Object.wait – Timed_Waiting. To use wait & notifyAll, you need to be clear about their performance ceilings and limitations. In JCIP (Java Concurrency in Practice) we also recommend to use the advanced concurrency primitive AQS (Abstract Queued Synchronizer) provided by JUC (Java Util Concurrency) as much as possible.
References:
1. http://architects.dzone.com/articles/how-analyze-java-thread-dumps
2. http://stackoverflow.com/questions/7698861/simple-java-example-runs-with-14-threads-why
3. http://docs.oracle.com/javase/1.5.0/docs/guide/misc/threadPrimitiveDeprecation.html
4. http://www.artima.com/insidejvm/ed2/threadsynch.html
5. http://stackoverflow.com/questions/37026/java-notify-vs-notifyall-all-over-again

Threads and Deadlock


A lock occurs when multiple processes try to access the same resource at the same time. A deadlock occurs when the waiting process is still holding on to another resource that the first needs before it can finish. A Java multithreaded program may suffer from the deadlock condition because the
synchronized keyword causes the executing thread to block while waiting
for the lock, or monitor , associated with the specified object.


More on....Deadlock in Java