HDFS-client may block shutdown of JVM
-------------------------------------
Key: HDFS-1868
URL: https://issues.apache.org/jira/browse/HDFS-1868
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.20.2
Environment: Linux 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25 EST
2011 x86_64 x86_64 x86_64 GNU/Linux
java version "1.6.0_23"
Java(TM) SE Runtime Environment (build 1.6.0_23-b05)
Java HotSpot(TM) 64-Bit Server VM (build 19.0-b09, mixed mode)
Reporter: Dennis Cheung
Priority: Minor
When I send a "kill -15" to shutdown my process, sometime it will freeze after
the safe-shutdown process complete.
I ran jstack and found the following thread blocking.
"SIGTERM handler" daemon prio=10 tid=0x00002aaab12c4000 nid=0x6274 waiting for
monitor entry [0x000000004e056000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Thread.join(Thread.java:1177)
- waiting to lock <0x000000078376e8b8> (a
org.apache.hadoop.fs.FileSystem$ClientFinalizer)
at java.lang.Thread.join(Thread.java:1239)
at
java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
at
java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.exit(Shutdown.java:168)
- locked <0x000000077e8127e8> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:35)
at sun.misc.Signal$1.run(Signal.java:195)
at java.lang.Thread.run(Thread.java:662)
Beside the thread "SIGTERM handler", there are some few threads of my
application are WAITING or BLOCKED with this pattern
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.writeChunk(DFSClient.java:3002)
- locked <0x00000007b24fee10> (a java.util.LinkedList)
- locked <0x00000007b2512070> (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:150)
at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:100)
at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:86)
- locked <0x00000007b2512070> (a
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream)
at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:49)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
- locked <0x00000007b25126e0> (a
org.apache.hadoop.fs.FSDataOutputStream)
However, there are no other HDFS threads (e.g. DataStreamer) running.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
