[
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284463#comment-16284463
]
Rohini Palaniswamy commented on PIG-5318:
-----------------------------------------
+1 on the patch from my side.
bq. found an universal way to tell the current Spark version
Did not suggest that as [~gezapeti] has mentioned earlier that it is
internal to Spark -
https://issues.apache.org/jira/browse/OOZIE-2606?focusedCommentId=15528793&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15528793.
It is ok here though as it is only for tests.
bq. testKeepGoingFailed is excluded because of SPARK-7953
SPARK-7953 still does not seem to be fixed as you mentioned. Can you try to
find which jira actually fixed it and probably close SPARK-7953 if it is not
required anymore. Identifying what behavior change caused this might also help
find other places in Pig on Spark that have to be fixed or changed for the new
behavior.
> Unit test failures on Pig on Spark with Spark 2.2
> -------------------------------------------------
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch,
> PIG-5318_4.patch, PIG-5318_5.patch, PIG-5318_6.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
> org.apache.pig.test.TestAssert#testNegativeWithoutFetch
> org.apache.pig.test.TestAssert#testNegative
> org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
> org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
> org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
> org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
> org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed
> by asserting on the message of the exception's root cause, looks like on
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
