[
https://issues.apache.org/jira/browse/HIVE-4318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13629247#comment-13629247
]
Kevin Wilfong commented on HIVE-4318:
-------------------------------------
It looks like the performance cost of those methods was recognized shortly
after they were added
https://issues.apache.org/jira/browse/HIVE-768
It's possible the config hive.task.progress isn't as effective at countering it
as expected.
Can you make sure it is set to false in your run.
> OperatorHooks hit performance even when not used
> ------------------------------------------------
>
> Key: HIVE-4318
> URL: https://issues.apache.org/jira/browse/HIVE-4318
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Environment: Ubuntu LXC (64 bit)
> Reporter: Gopal V
> Assignee: Gunther Hagleitner
> Attachments: HIVE-4318.1.patch
>
>
> Operator Hooks inserted into Operator.java cause a performance hit even when
> it is not being used.
> For a count(1) query tested with & without the operator hook calls.
> {code:title=with}
> 2013-04-09 07:33:58,920 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
> 84.07 sec
> Total MapReduce CPU Time Spent: 1 minutes 24 seconds 70 msec
> OK
> 28800991
> Time taken: 40.407 seconds, Fetched: 1 row(s)
> {code}
> {code:title=without}
> 2013-04-09 07:33:02,355 Stage-1 map = 100%, reduce = 100%, Cumulative CPU
> 68.48 sec
> ...
> Total MapReduce CPU Time Spent: 1 minutes 8 seconds 480 msec
> OK
> 28800991
> Time taken: 35.907 seconds, Fetched: 1 row(s)
> {code}
> The effect is multiplied by the number of operators in the pipeline that has
> to forward the row - the more operators there are the, the slower the query.
> The modification made to test this was
> {code:title=Operator.java}
> --- ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
> +++ ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
> @@ -526,16 +526,16 @@ public void process(Object row, int tag) throws
> HiveException {
> return;
> }
> OperatorHookContext opHookContext = new OperatorHookContext(this, row,
> tag);
> - preProcessCounter();
> - enterOperatorHooks(opHookContext);
> + //preProcessCounter();
> + //enterOperatorHooks(opHookContext);
> processOp(row, tag);
> - exitOperatorHooks(opHookContext);
> - postProcessCounter();
> + //exitOperatorHooks(opHookContext);
> + //postProcessCounter();
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
