[
https://issues.apache.org/jira/browse/HIVE-11681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15853425#comment-15853425
]
muxin commented on HIVE-11681:
------------------------------
hadoop-mapreduce-client-core.jar is loaded when starting hiveserver2, so the
specific classloader hold this jar should be parent of classloaders in
'org.apache.hadoop.hive.common.JavaUtils.closeClassLoadersTo(JavaUtils.java:79)'.
so how could it be closed when calling closeClassLoadersTo function? when
adding a hdfs jar, hiveserver2 will download it into
${hive.session.id}_resources folder, which is different among sessions, is this
condition JarURLConnection is different too. Only when adding a local jar
SessionState directly add the path to classpath, so different sessions could
hold the same connection of it, and then cause ioexception as mentioned. I
don't get it how does it cause ioexception on hadoop-mapreduce-client-core.jar
> sometimes when query mr job progress, stream closed exception will happen
> -------------------------------------------------------------------------
>
> Key: HIVE-11681
> URL: https://issues.apache.org/jira/browse/HIVE-11681
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 1.2.1
> Reporter: wangwenli
>
> sometimes the hiveserver will throw below exception ,
> 2015-08-28 05:05:44,107 | FATAL | Thread-82995 | error parsing conf
> mapred-default.xml |
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2404)
> java.io.IOException: Stream closed
> at
> java.util.zip.InflaterInputStream.ensureOpen(InflaterInputStream.java:84)
> at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:160)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2902)
> at
> com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:302)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at
> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2246)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2234)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2305)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2258)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2175)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:854)
> at
> org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2069)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:477)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:467)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:187)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1612)
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578)
> at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:596)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:289)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:548)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:435)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
> after analysis, we found the root cause, below is step to reproduce the issue
> 1. open one beeline window, add jar
> 2. execute one sql like create table abc as select * from t1;
> 3. execute !quit, before this add a breakpoint at
> java.net.URLClassLoader.close(), so it will stop here
> 4. open another beeline window
> 5. execute one sql like select count(*) from t1, before this add one
> condition breakpoint at org.apache.hadoop.conf.Configuration.parse, the
> condition is
> url.toString().indexOf("hadoop-mapreduce-client-core-V100R001C00.jar!/mapred-default.xml")>0
> 6. when proceed to the above step, just get the stream
> 7. let the 3th step go ahead, which will close the stream
> 8. now, let sixth step go ahead, then the above exception coming
> suggest solution:
> the issue is hadppend when client use add jar + short connection , the
> client app should try best to use long connection, means reuse hive connetion
> meanwhile, here ,experts can suggest good solution for this issue
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
