Istvan Szukacs created HDFS-7212:
------------------------------------
Summary: Huge number of BLOCKED threads rendering DataNodes useless
Key: HDFS-7212
URL: https://issues.apache.org/jira/browse/HDFS-7212
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.4.0
Environment: PROD
Reporter: Istvan Szukacs
There are 3000 - 8000 threads in each datanode JVM, blocking the entire VM and
rendering the service unusable, missing heartbeats and stopping data access.
The threads look like this:
{code}
3415 (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=186 (Compiled frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt()
@bci=1, line=834 (Interpreted frame)
-
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node,
int) @bci=67, line=867 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17,
line=1197 (Interpreted frame)
- java.util.concurrent.locks.ReentrantLock$NonfairSync.lock() @bci=21, line=214
(Compiled frame)
- java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=290 (Compiled
frame)
-
org.apache.hadoop.net.unix.DomainSocketWatcher.add(org.apache.hadoop.net.unix.DomainSocket,
org.apache.hadoop.net.unix.DomainSocketWatcher$Handler) @bci=4, line=286
(Interpreted frame)
-
org.apache.hadoop.hdfs.server.datanode.ShortCircuitRegistry.createNewMemorySegment(java.lang.String,
org.apache.hadoop.net.unix.DomainSocket) @bci=169, line=283 (Interpreted frame)
-
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitShm(java.lang.String)
@bci=212, line=413 (Interpreted frame)
-
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitShm(java.io.DataInputStream)
@bci=13, line=172 (Interpreted frame)
-
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(org.apache.hadoop.hdfs.protocol.datatransfer.Op)
@bci=149, line=92 (Compiled frame)
- org.apache.hadoop.hdfs.server.datanode.DataXceiver.run() @bci=510, line=232
(Compiled frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
{code}
Has anybody seen this before?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
