Chao Sun created HIVE-15659:
-------------------------------
Summary: StackOverflowError when ClassLoader.loadClass for Spark
Key: HIVE-15659
URL: https://issues.apache.org/jira/browse/HIVE-15659
Project: Hive
Issue Type: Bug
Components: Spark
Affects Versions: 2.2.0
Reporter: Chao Sun
Sometimes a query needs to process a large number of input files, which could
cause the following error:
{code}
17/01/15 09:31:52 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0
(TID 0, hadoopworker1344-sjc1.prod.uber.internal): java.lang.StackOverflowError
at
java.util.concurrent.ConcurrentHashMap.putIfAbsent(ConcurrentHashMap.java:1535)
at java.lang.ClassLoader.getClassLoadingLock(ClassLoader.java:463)
at java.lang.ClassLoader.loadClass(ClassLoader.java:404)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
{code}
The cause, I think, is that for each input file we may need to load additional
jars to the class loader of the current thread. This accumulates with the
number of input files. When adding a new class loader, the old class loader
will be used as the parent of the new one.
See
[Utilities#getBaseWork|https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L388]
for more details.
One possible solution is to detect duplicated jar paths before creating the new
class loader.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
