[
https://issues.apache.org/jira/browse/HIVE-4518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13819262#comment-13819262
]
Jason Dere commented on HIVE-4518:
----------------------------------
[~navis], took a look at this patch yesterday and was thinking that this patch
could have been done without completely getting rid of hive.task.progress mode,
looks like we could have made changes such that hive.task.progress wasn't
required when doing dynamic partitioning. Anyway, if you've already gone
through the trouble to rebase this patch we can go ahead with this as it is.
> Counter Strike: Operation Operator
> ----------------------------------
>
> Key: HIVE-4518
> URL: https://issues.apache.org/jira/browse/HIVE-4518
> Project: Hive
> Issue Type: Improvement
> Reporter: Gunther Hagleitner
> Assignee: Gunther Hagleitner
> Attachments: HIVE-4518.1.patch, HIVE-4518.2.patch, HIVE-4518.3.patch,
> HIVE-4518.4.patch, HIVE-4518.5.patch, HIVE-4518.6.patch.txt
>
>
> Queries of the form:
> from foo
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> insert overwrite table bar partition (p) select ...
> Generate a huge amount of counters. The reason is that task.progress is
> turned on for dynamic partitioning queries.
> The counters not only make queries slower than necessary (up to 50%) you will
> also eventually run out. That's because we're wrapping them in enum values to
> comply with hadoop 0.17.
> The real reason we turn task.progress on is that we need CREATED_FILES and
> FATAL counters to ensure dynamic partitioning queries don't go haywire.
> The counters have counter-intuitive names like C1 through C1000 and don't
> seem really useful by themselves.
> With hadoop 20+ you don't need to wrap the counters anymore, each operator
> can simply create and increment counters. That should simplify the code a lot.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
