[
https://issues.apache.org/jira/browse/HIVE-7955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14202958#comment-14202958
]
Thomas Friedrich commented on HIVE-7955:
----------------------------------------
The test hook_context_cs fails while checking for expected value in
VerifyContentSummaryCacheHook:
TestSparkCliDriver.testCliDriver_hook_context_cs:120->runTest:146 Unexpected
exception junit.framework.AssertionFailedError: expected:<1> but was:<0>
at junit.framework.Assert.fail(Assert.java:57)
at junit.framework.Assert.failNotEquals(Assert.java:329)
at junit.framework.Assert.assertEquals(Assert.java:78)
at junit.framework.Assert.assertEquals(Assert.java:234)
at junit.framework.Assert.assertEquals(Assert.java:241)
at
org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook.run(VerifyContentSummaryCacheHook.java:34)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1512)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1216)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1043)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1033)
The problem is that the post-hook check in
org.apache.hadoop.hive.ql.hooks.VerifyContentSummaryCacheHook fails because
inputToCS.size is 0 instead of 1:
Assert.assertEquals(1, inputToCS.size());
I ran the same test with TestCliDriver where Hive will create MapRedTask
objects. For a MapRedTask, the hookContext.getInputPathToContentSummary() is
set in the setNumberOfReducers method, line 400:
inputSummary = Utilities.getInputSummary(driverContext.getCtx(),
work.getMapWork(), null);
(the path is added in Utilities.getInputSummary(Context, MapWork, PathFilter)
line: 2584).
Since we have a SparkTask, the array will be empty, hence the check for
length==1 fails.
It looks somewhat related to HIVE-8504 where the test case also configures a
hive.exec.post.hooks, but since it's a different post-hook class altogether, I
opened a separate JIRA HIVE-8790.
Maybe something needs to be added to the SparkTask.
> Investigate query failures (4)
> ------------------------------
>
> Key: HIVE-7955
> URL: https://issues.apache.org/jira/browse/HIVE-7955
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Reporter: Brock Noland
> Assignee: Thomas Friedrich
>
> I ran all q-file tests and the following failed with an exception:
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/HIVE-SPARK-ALL-TESTS-Build/lastCompletedBuild/testReport/
> we don't necessary want to run all these tests as part of the spark tests,
> but we should understand why they failed with an exception. This JIRA is to
> look into these failures and document them with one of:
> * New JIRA
> * Covered under existing JIRA
> * More investigation required
> Tests:
> {noformat}
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dynpart_sort_optimization
> 12 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_schemeAuthority2
> 0.23 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_load_dyn_part8
> 10 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_4
> 11 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_orc_analyze
> 8 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_tez_join_hash
> 0.98 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_hook_context_cs
> 2.1 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_insert_overwrite_local_directory_1
> 3.7 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_archive_excludeHadoop20
> 27 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_9
> 8.2 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_partition_metadataonly
> 0.77 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucket_num_reducers2
> 7 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_groupby_bigdata
> 0.6 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_bucketsortoptimize_insert_6
> 6.6 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_25
> 2.6 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_dbtxnmgr_query3
> 0.48 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_16
> 8.5 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_empty_dir_in_table
> 2.6 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_input33 1.3 sec
> 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_authorization_admin_almighty1
> 2.8 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_udf_context_aware
> 0.23 sec 2
>
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_authorization_view_sqlstd
> 4.1 sec 2
> org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_smb_mapjoin_12
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
