[
https://issues.apache.org/jira/browse/HIVE-15114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15633826#comment-15633826
]
Sergio Peña commented on HIVE-15114:
------------------------------------
Disabling the merge ({{hive.merge.mapfiles=false}}) removes the extra MoveTask
operator from the query. Thanks [~sershe] for the tip.
However, I think there is still another move (based on the 'hdfs dfs -ls s3a'
output) besides the FileSinkOperator.
The below output was taking listing the directories that contain 00000_0 every
second. Don't take the below output as just one ls.
WITHOUT MERGE (2 renames)
{noformat}
-rw-rw-rw- 1 2 2016-11-03 13:22
s3a://spena-bucket/s3test/.hive-staging_hive_2016-11-03_13-22-41_807_3638905212724497364-1/_tmp.-ext-10000/000000_0
-rw-rw-rw- 1 2 2016-11-03 13:23
s3a://spena-bucket/s3test/.hive-staging_hive_2016-11-03_13-22-41_807_3638905212724497364-1/-ext-10000/000000_0
-rw-rw-rw- 1 2 2016-09-26 16:45 s3a://spena-bucket/s3test/000000_0
{noformat}
WITH MERGE (3 renames)
{noformat}
-rw-rw-rw- 1 2 2016-11-03 13:16
s3a://spena-bucket/s3test/.hive-staging_hive_2016-11-03_13-15-56_254_5576519570216642148-1/_tmp.-ext-10002/000000_0
-rw-rw-rw- 1 2 2016-11-03 13:16
s3a://spena-bucket/s3test/.hive-staging_hive_2016-11-03_13-15-56_254_5576519570216642148-1/-ext-10002/000000_0
-rw-rw-rw- 1 2 2016-11-03 13:16
s3a://spena-bucket/s3test/.hive-staging_hive_2016-11-03_13-15-56_254_5576519570216642148-1/-ext-10000/000000_0
-rw-rw-rw- 1 2 2016-09-26 16:45 s3a://spena-bucket/s3test/000000_0
{noformat}
> Remove extra MoveTask operators
> -------------------------------
>
> Key: HIVE-15114
> URL: https://issues.apache.org/jira/browse/HIVE-15114
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
> Affects Versions: 2.1.0
> Reporter: Sahil Takiar
> Assignee: Sergio Peña
>
> When running simple insert queries (e.g. {{INSERT INTO TABLE ... VALUES
> ...}}) there an extraneous {{MoveTask}s is created.
> This is problematic when the scratch directory is on S3 since renames require
> copying the entire dataset.
> For simple queries (like the one above), there are two MoveTasks. The first
> one moves the output data from one file in the scratch directory to another
> file in the scratch directory. The second MoveTask moves the data from the
> scratch directory to its final table location.
> The first MoveTask should not be necessary. The goal of this JIRA it to
> remove it. This should help improve performance when running on S3.
> It seems that the first Move might be caused by a dependency resolution
> problem in the optimizer, where a dependent task doesn't get properly removed
> when the task it depends on is filtered by a condition resolver.
> A dummy {{MoveTask}} is added in the
> {{GenMapRedUtils.createMRWorkForMergingFiles}} method. This method creates a
> conditional task which launches a job to merge tasks at the end of the file.
> At the end of the conditional job there is a MoveTask.
> Even though Hive decides that the conditional merge job is no needed, it
> seems the MoveTask is still added to the plan.
> Seems this extra {{MoveTask}} may have been added intentionally. Not sure why
> yet. The {{ConditionalResolverMergeFiles}} says that one of three tasks will
> be returned: move task only, merge task only, merge task followed by a move
> task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
