[
https://issues.apache.org/jira/browse/HIVE-12161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955749#comment-14955749
]
Sergey Shelukhin commented on HIVE-12161:
-----------------------------------------
[~vikram.dixit] [~sseth] fyi.
I tried to do some analysis on builds 5618 (before), 5629 (after) and 5626 (AM
reuse patch).
Then I realized we have 5596 build results which is AM reuse before merge and
it also took 2 hours and correlation is 0.98 in runtimes. So I think this is
entirely attributable to session setup.
The test time differences between 5618 and 5629 are don't have any discernible
pattern (e.g. (before-after-after+AM reuse) a few tests even got faster
testCliDriver_create_merge_compressed 41 36 17 , some tests got
many minutes slower, testCliDriver_auto_sortmerge_join_12 23 299
5.3, or testCliDriver_load_dyn_part1 27 321 3.5 294).
Average benefit from session reuse "before" is 24sec, 8sec. deviation, average
benefit "after" is 155sec., with 118sec. stdev.
Should we investigate why session setup is so slow and random? Will it affect
real clusters?
> MiniTez test is very slow since LLAP branch merge
> -------------------------------------------------
>
> Key: HIVE-12161
> URL: https://issues.apache.org/jira/browse/HIVE-12161
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
>
> Before merge, the test took 4~hrs (total time parallelized, not wall clock
> time), after the merge it's taking 12-15hrs. First such build:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5622/testReport/org.apache.hadoop.hive.cli/TestMiniTezCliDriver/
> -Session reuse patch which used to make them super fast now makes them run in
> 2hrs-
> -http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/5628/testReport/org.apache.hadoop.hive.cli/TestMiniTezCliDriver/
> which is still a lot.- This is an invalid statement
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
