[
https://issues.apache.org/jira/browse/HIVE-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13428782#comment-13428782
]
Hudson commented on HIVE-3302:
------------------------------
Integrated in Hive-trunk-h0.21 #1592 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/1592/])
HIVE-3302 Race condition in query plan for merging at the end of a query
(Kevin Wilfong via namit) (Revision 1369375)
Result = SUCCESS
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369375
Files :
* /hive/trunk/contrib/src/test/results/clientpositive/serde_typedbytes.q.out
* /hive/trunk/contrib/src/test/results/clientpositive/serde_typedbytes2.q.out
* /hive/trunk/contrib/src/test/results/clientpositive/serde_typedbytes3.q.out
* /hive/trunk/contrib/src/test/results/clientpositive/serde_typedbytes5.q.out
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
* /hive/trunk/ql/src/test/queries/clientpositive/merge_dynamic_partition5.q
* /hive/trunk/ql/src/test/results/clientpositive/binary_output_format.q.out
* /hive/trunk/ql/src/test/results/clientpositive/bucketmapjoin1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/bucketmapjoin2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/bucketmapjoin3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/bucketmapjoin4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/bucketmapjoin5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out
* /hive/trunk/ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/case_sensitivity.q.out
* /hive/trunk/ql/src/test/results/clientpositive/cast1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/index_auto.q.out
* /hive/trunk/ql/src/test/results/clientpositive/index_auto_file_format.q.out
*
/hive/trunk/ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out
* /hive/trunk/ql/src/test/results/clientpositive/index_auto_multiple.q.out
* /hive/trunk/ql/src/test/results/clientpositive/index_auto_partitioned.q.out
* /hive/trunk/ql/src/test/results/clientpositive/index_auto_update.q.out
* /hive/trunk/ql/src/test/results/clientpositive/index_compression.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input11.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input12.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input13.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input34.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input35.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input36.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input38.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input6.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input7.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input8.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input9.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input_dynamicserde.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input_part1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input_part2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input_part5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input_testsequencefile.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input_testxpath.q.out
* /hive/trunk/ql/src/test/results/clientpositive/input_testxpath2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/insert1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/insert_into4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/insert_into5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/insert_into6.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join25.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join26.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join27.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join28.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join29.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join32.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join34.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join35.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join36.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join37.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join39.q.out
* /hive/trunk/ql/src/test/results/clientpositive/join_map_ppr.q.out
* /hive/trunk/ql/src/test/results/clientpositive/lineage1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/load_dyn_part1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/load_dyn_part14.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge_dynamic_partition.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge_dynamic_partition2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge_dynamic_partition3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge_dynamic_partition4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/merge_dynamic_partition5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/multi_insert.q.out
*
/hive/trunk/ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out
* /hive/trunk/ql/src/test/results/clientpositive/pcr.q.out
* /hive/trunk/ql/src/test/results/clientpositive/ppd_constant_expr.q.out
* /hive/trunk/ql/src/test/results/clientpositive/quote1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/rand_partitionpruner2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/rcfile_createas1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/rcfile_merge1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/rcfile_merge2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/rcfile_merge3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/rcfile_merge4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/sample1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/sample2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/sample4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/sample5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/sample6.q.out
* /hive/trunk/ql/src/test/results/clientpositive/sample7.q.out
* /hive/trunk/ql/src/test/results/clientpositive/stats0.q.out
* /hive/trunk/ql/src/test/results/clientpositive/stats11.q.out
* /hive/trunk/ql/src/test/results/clientpositive/stats4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/subq.q.out
* /hive/trunk/ql/src/test/results/clientpositive/udf1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/udf_10_trims.q.out
* /hive/trunk/ql/src/test/results/clientpositive/udf_length.q.out
* /hive/trunk/ql/src/test/results/clientpositive/udf_reverse.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union10.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union12.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union18.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union28.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union29.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union30.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union6.q.out
* /hive/trunk/ql/src/test/results/compiler/plan/case_sensitivity.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/input1.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/input2.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/input3.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/input6.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/input7.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/input9.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/input_testsequencefile.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/sample2.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/sample3.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/sample4.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/sample5.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/sample6.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/sample7.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/subq.q.xml
* /hive/trunk/ql/src/test/results/compiler/plan/union.q.xml
> Race condition in query plan for merging at the end of a query
> --------------------------------------------------------------
>
> Key: HIVE-3302
> URL: https://issues.apache.org/jira/browse/HIVE-3302
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 0.10.0
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Priority: Critical
> Attachments: HIVE-3302.1.patch.txt, HIVE-3302.2.patch.txt
>
>
> In the query plan that's used to merge files at the end of a query, the
> dependency tree looks something like:
> MoveTask(1)
> / \
> ...ConditionalTask MoveTask(2)...
> \ /
> MergeTask
> Here MoveTask(1) moves the partition data to a temporary location, and
> MoveTask(2) moves it to the final location.
> However if there are dynamic partitions generated and some of these
> partitions are merged and others are moved, the dependency tree is changed at
> runtime to:
> ...ConditionalTask MoveTask(2)...
> \ /
> MergeTask
> \
> MoveTask(1)
> This produces a race condition between the two MoveTasks where if MoveTask(2)
> runs before MoveTask(1) the partitions moved by MoveTask(1) will get moved to
> an intermediate location and never moved to the final location. In this case
> those partitions are quietly lost.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
