[
https://issues.apache.org/jira/browse/HIVE-11681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736752#comment-14736752
]
wangwenli commented on HIVE-11681:
----------------------------------
below is the stacktrace that stream be cached in the URLClassloader when load
TaskCounter.properties in hadoop-mapreduce-client-core.jar
Thread [Thread-31585] (Suspended (breakpoint at line 239 in URLClassLoader))
owns: WeakHashMap<K,V> (id=2282)
owns: Class<T> (org.apache.hadoop.mapreduce.util.ResourceBundles)
(id=1456)
owns: Counters (id=2283)
URLClassLoader.getResourceAsStream(String) line: 239
ResourceBundle$Control$1.run() line: 2601 [local variables unavailable]
ResourceBundle$Control$1.run() line: 2586
AccessController.doPrivileged(PrivilegedExceptionAction<T>) line: not
available [native method]
ResourceBundle$Control.newBundle(String, Locale, String, ClassLoader,
boolean) line: 2585
ResourceBundle.loadBundle(CacheKey, List<String>, Control, boolean)
line: 1436
ResourceBundle.findBundle(CacheKey, List<Locale>, List<String>, int,
Control, ResourceBundle) line: 1400
ResourceBundle.findBundle(CacheKey, List<Locale>, List<String>, int,
Control, ResourceBundle) line: 1354
ResourceBundle.findBundle(CacheKey, List<Locale>, List<String>, int,
Control, ResourceBundle) line: 1354
ResourceBundle.getBundleImpl(String, Locale, ClassLoader,
ResourceBundle$Control) line: 1296
ResourceBundle.getBundle(String, Locale, ClassLoader) line: 1028
ResourceBundles.getBundle(String) line: 37
ResourceBundles.getValue(String, String, String, T) line: 56
ResourceBundles.getCounterGroupName(String, String) line: 77
Counters$GroupFactory(CounterGroupFactory<C,G>).newGroup(String,
Limits) line: 94
Counters(AbstractCounters<C,G>).getGroup(String) line: 226
Counters.getGroup(String) line: 113
Counters.findCounter(String, String) line: 479
HadoopJobExecHelper.progress(HadoopJobExecHelper$ExecDriverTaskHandle)
line: 347
HadoopJobExecHelper.progress(RunningJob, JobClient, HiveTxnManager)
line: 548
MapRedTask(ExecDriver).execute(DriverContext) line: 435
MapRedTask.execute(DriverContext) line: 159
MapRedTask(Task<T>).executeTask() line: 153
TaskRunner.runSequential() line: 85
TaskRunner.run() line: 72
below is the stacktrace that steam is closed:
java.util.zip.InflaterInputStream.getStack(InflaterInputStream.java:309)
java.util.zip.InflaterInputStream.close(InflaterInputStream.java:247)
java.util.zip.ZipFile$ZipFileInflaterInputStream.close(ZipFile.java:398)
java.util.zip.ZipFile.close(ZipFile.java:585)
sun.net.www.protocol.jar.URLJarFile.close(URLJarFile.java:167)
java.net.URLClassLoader.close(URLClassLoader.java:294)
org.apache.hadoop.hive.common.JavaUtils.closeClassLoader(JavaUtils.java:101)
org.apache.hadoop.hive.common.JavaUtils.closeClassLoadersTo(JavaUtils.java:79)
org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:942)
org.apache.hive.service.cli.session.HiveSessionImpl.close(HiveSessionImpl.java:562)
so here the root cause is that: add jar will lead to new classloader created,
and it will cache the JarFile , but JarFile in jvm only has one instance(see
JarURLConnection.setUseCache), so when other thread is read from the same
JarFile, once hive session closed will lead to classloader close the stream.
> sometimes when query mr job progress, stream closed exception will happen
> -------------------------------------------------------------------------
>
> Key: HIVE-11681
> URL: https://issues.apache.org/jira/browse/HIVE-11681
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 1.2.1
> Reporter: wangwenli
>
> sometimes the hiveserver will throw below exception ,
> 2015-08-28 05:05:44,107 | FATAL | Thread-82995 | error parsing conf
> mapred-default.xml |
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2404)
> java.io.IOException: Stream closed
> at
> java.util.zip.InflaterInputStream.ensureOpen(InflaterInputStream.java:84)
> at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:160)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityManager$RewindableInputStream.read(XMLEntityManager.java:2902)
> at
> com.sun.org.apache.xerces.internal.impl.io.UTF8Reader.read(UTF8Reader.java:302)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.load(XMLEntityScanner.java:1753)
> at
> com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.skipChar(XMLEntityScanner.java:1426)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2807)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)
> at
> com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:117)
> at
> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)
> at
> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)
> at
> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
> at
> com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:243)
> at
> com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:347)
> at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2246)
> at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2234)
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2305)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2258)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2175)
> at org.apache.hadoop.conf.Configuration.get(Configuration.java:854)
> at
> org.apache.hadoop.mapred.JobConf.checkAndWarnDeprecation(JobConf.java:2069)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:477)
> at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:467)
> at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:187)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:580)
> at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:578)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1612)
> at
> org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:578)
> at org.apache.hadoop.mapred.JobClient.getJob(JobClient.java:596)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:289)
> at
> org.apache.hadoop.hive.ql.exec.mr.HadoopJobExecHelper.progress(HadoopJobExecHelper.java:548)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:435)
> at
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:159)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:72)
> after analysis, we found the root cause, below is step to reproduce the issue
> 1. open one beeline window, add jar
> 2. execute one sql like create table abc as select * from t1;
> 3. execute !quit, before this add a breakpoint at
> java.net.URLClassLoader.close(), so it will stop here
> 4. open another beeline window
> 5. execute one sql like select count(*) from t1, before this add one
> condition breakpoint at org.apache.hadoop.conf.Configuration.parse, the
> condition is
> url.toString().indexOf("hadoop-mapreduce-client-core-V100R001C00.jar!/mapred-default.xml")>0
> 6. when proceed to the above step, just get the stream
> 7. let the 3th step go ahead, which will close the stream
> 8. now, let sixth step go ahead, then the above exception coming
> suggest solution:
> the issue is hadppend when client use add jar + short connection , the
> client app should try best to use long connection, means reuse hive connetion
> meanwhile, here ,experts can suggest good solution for this issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
