Chao Sun created HIVE-16175:
-------------------------------
Summary: Possible race condition in InstanceCache
Key: HIVE-16175
URL: https://issues.apache.org/jira/browse/HIVE-16175
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Reporter: Chao Sun
Assignee: Chao Sun
Currently the [InstanceCache |
https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/avro/InstanceCache.java]
class is not thread safe, but it is sometimes used as a static variable, for
instance
[here|https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/avro/SchemaToTypeInfo.java#L114].
This will be an issue for HoS, at which scenario it could be accessed by
multiple threads at the same time. We found this sometimes causes NPE:
{code}
ERROR : FAILED: Execution Error, return code 3 from
org.apache.hadoop.hive.ql.exec.spark.SparkTask.
java.util.concurrent.ExecutionException: Exception thrown by job
at
org.apache.spark.JavaFutureActionWrapper.getImpl(FutureAction.scala:311)
at org.apache.spark.JavaFutureActionWrapper.get(FutureAction.scala:316)
at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:382)
at
org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:335)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure:
Task 20 in stage 0.0 failed 4 times, most recent failure: Lost task 20.3 in
stage 0.0 (TID 33, hadoopworker992-sjc1.prod.uber.internal):
java.lang.RuntimeException: Map operator initialization failed:
org.apache.hadoop.hive.ql.metadata.Hive
Exception: java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:127)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:55)
at
org.apache.hadoop.hive.ql.exec.spark.HiveMapFunction.call(HiveMapFunction.java:30)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
at
org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$7$1.apply(JavaRDDLike.scala:192)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at
org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$20.apply(RDD.scala:710)
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:270)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:325)
at
org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:388)
at
org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.init(SparkMapRecordHandler.java:92)
... 16 more
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.supportedCategories(AvroObjectInspectorGenerator.java:142)
at
org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:91)
at
org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:104)
at
org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:104)
at
org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspectorWorker(AvroObjectInspectorGenerator.java:121)
at
org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.createObjectInspector(AvroObjectInspectorGenerator.java:83)
at
org.apache.hadoop.hive.serde2.avro.AvroObjectInspectorGenerator.<init>(AvroObjectInspectorGenerator.java:56)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:126)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:80)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:520)
at
org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:142)
at
org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:299)
... 18 more
{code}
(The stack trace is from Hive 1.1.0 but the issue should also exist in
upstream).
This issue is also reported earlier:
https://groups.google.com/forum/#!topic/presto-users/CpEDvLiUSX0
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
