Hive JDBC Server: java.lang.IllegalStateException: Shutdown in progress

[Guy Doulberg]

Hi guys,

I have a hive JDBC server in production,
It started lately to fail.
In the log files I can see the following:

2013-09-02_10:42:53.13215 java.lang.IllegalStateException: Shutdown in 
progress, cannot add a shutdownHook
2013-09-02_10:42:53.13215       at 
org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152)
2013-09-02_10:42:53.13216       at 
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2341)
2013-09-02_10:42:53.13216       at 
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2313)
2013-09-02_10:42:53.13217       at 
org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
2013-09-02_10:42:53.13217       at 
org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
2013-09-02_10:42:53.13219       at 
org.apache.hadoop.hive.ql.exec.Utilities.realFile(Utilities.java:1027)
2013-09-02_10:42:53.13219       at 
org.apache.hadoop.hive.ql.exec.Utilities.getResourceFiles(Utilities.java:1551)
2013-09-02_10:42:53.13220       at 
org.apache.hadoop.hive.ql.exec.ExecDriver.initialize(ExecDriver.java:152)
2013-09-02_10:42:53.13220       at 
org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1343)
2013-09-02_10:42:53.13221       at 
org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1138)
2013-09-02_10:42:53.13221       at 
org.apache.hadoop.hive.ql.Driver.run(Driver.java:951)
2013-09-02_10:42:53.13222       at 
org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:198)
2013-09-02_10:42:53.13224       at 
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:644)
2013-09-02_10:42:53.13224       at 
org.apache.hadoop.hive.service.ThriftHive$Processor$execute.getResult(ThriftHive.java:628)
2013-09-02_10:42:53.13225       at 
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
2013-09-02_10:42:53.13225       at 
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
2013-09-02_10:42:53.13226       at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:206)
2013-09-02_10:42:53.13226       at 
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
2013-09-02_10:42:53.13227       at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
2013-09-02_10:42:53.13227       at java.lang.Thread.run(Thread.java:662)
2013-09-02_10:42:53.13228
2013-09-02_10:42:53.13761 FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.MapRedTask
2013-09-02_10:42:53.13763 FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.MapRedTask


Has some one encoutered this problem, and know why it is happening?

The jmx doesn't expose anything interesting.


Guy