[
https://issues.apache.org/jira/browse/HIVE-8913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14242148#comment-14242148
]
Chao commented on HIVE-8913:
----------------------------
Looking at the log, something interesting..
{noformat}
2014-12-10 05:28:26,822 INFO [Executor task launch worker-0]:
exec.GroupByOperator (Operator.java:close(613)) - 44 Close done
2014-12-10 05:28:26,822 INFO [Executor task launch worker-0]:
exec.FilterOperator (Operator.java:close(613)) - 43 Close done
2014-12-10 05:28:26,822 INFO [Executor task launch worker-0]:
exec.ForwardOperator (Operator.java:close(613)) - 38 Close done
2014-12-10 05:28:26,828 INFO [Executor task launch worker-1]:
executor.Executor (Logging.scala:logInfo(59)) - Finished task 2.0 in stage 4.0
(TID 6). 1170 bytes result sent to driver
2014-12-10 05:28:26,828 INFO [Executor task launch worker-0]:
executor.Executor (Logging.scala:logInfo(59)) - Finished task 1.0 in stage 4.0
(TID 5). 1170 bytes result sent to driver
....
2014-12-10 05:28:26,979 INFO [Executor task launch worker-0]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(135)) - </PERFLOG method=deserializePlan
start=1418218106762 end=1418218106979 duration=217
from=org.apache.hadoop.hive.ql.exec.Utilities>
2014-12-10 05:28:26,980 INFO [Executor task launch worker-0]: mr.ObjectCache
(ObjectCache.java:cache(36)) - Ignoring cache key: __MAP_PLAN__
2014-12-10 05:28:26,982 ERROR [Executor task launch worker-0]:
executor.Executor (Logging.scala:logError(96)) - Exception in task 0.0 in stage
2.0 (TID 2)
java.lang.RuntimeException: Map operator initialization failed
{noformat}
At first, worker-0 is running a reduce work. After finishing that, it is
switched to this ShuffleMapTask, which apparently doesn't go through
{{HadoopRDD#compute}}, and thus IOContext is left uninitialized.
> Make SparkMapJoinResolver handle runtime skew join [Spark Branch]
> -----------------------------------------------------------------
>
> Key: HIVE-8913
> URL: https://issues.apache.org/jira/browse/HIVE-8913
> Project: Hive
> Issue Type: Improvement
> Components: Spark
> Reporter: Rui Li
> Assignee: Rui Li
> Attachments: HIVE-8913.1-spark.patch, HIVE-8913.2-spark.patch
>
>
> Sub-task of HIVE-8406.
> Now we have {{SparkMapJoinResolver}} in place. But at the moment, it doesn't
> handle the map join task created by upstream SkewJoinResolver, i.e. those
> wrapped in a ConditionalTask. We have to implement this part for runtime skew
> join to work on spark. To do so, we can borrow logic from {{MapJoinResolver}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
