[
https://issues.apache.org/jira/browse/PIG-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280419#comment-16280419
]
Nandor Kollar commented on PIG-5318:
------------------------------------
[~rohini] I agree, this looks like a hacky solution, but I couldn't think of a
better one, I'll try to figure out something better.
bq. It is hacky and will break Pig local mode and Tez.
Are you saying, that if I add a test case which stores in two different stores,
similar to the existing one in TestStoreInstances but with two STORE
statements, then that should pass too? I tried this:
{code}
@Test
public void testBackendMultiStoreCommunication() throws Exception {
ExecType[] execTypes = { Util.getLocalTestMode() };
PigServer pig = null;
for(ExecType execType : execTypes){
Util.resetStateForExecModeSwitch();
System.err.println("Starting test mode " + execType);
if (execType == cluster.getExecType()) {
pig = new PigServer(cluster.getExecType(),
cluster.getProperties());
} else {
pig = new PigServer(execType);
}
final String outFile = "TestStoreInst1";
final String outFile2 = "TestStoreInst2";
Util.deleteFile(pig.getPigContext(), outFile);
Util.deleteFile(pig.getPigContext(), outFile2);
pig.setBatchOn();
String query =
" l1 = load '" + INP_FILE_2NUMS + "' as (i : int, j :
int);" +
" store l1 into '" + outFile + "' using " +
CHECK_INSTANCE_STORE_FUNC +
";" +
" store l1 into '" + outFile2 + "' using " +
CHECK_INSTANCE_STORE_FUNC +
";";
Util.registerMultiLineQuery(pig, query);
List<ExecJob> execJobs = pig.executeBatch();
assertEquals("num jobs", 2, execJobs.size());
assertEquals("status ", JOB_STATUS.COMPLETED,
execJobs.get(0).getStatus());
}
}
{code}
But it failed in local mode.
bq. Can you make it isSpark2_2_plus which is slightly more intuitive than
2_1_minus. Also instantiating SparkContext just to get version seems overkill.
Ok, I'll revert to the original option, and not instantiate SparkContext
(agree, it is an overkill, that's why I tried an other solution first), though
I think that's the most reliable way to tell the Spark version, leaving the
details for Spark code. Checking for {{spark-version-info.properties}} looks
very version dependent solution.
Should we open a separate Jira for fixing TestStoreInstances in spark mode? It
seems that the other items are straightforward, and more test related issues,
but this one seems to be something to fix in code instead of in test.
> Unit test failures on Pig on Spark with Spark 2.2
> -------------------------------------------------
>
> Key: PIG-5318
> URL: https://issues.apache.org/jira/browse/PIG-5318
> Project: Pig
> Issue Type: Bug
> Components: spark
> Reporter: Nandor Kollar
> Assignee: Nandor Kollar
> Attachments: PIG-5318_1.patch, PIG-5318_2.patch, PIG-5318_3.patch,
> PIG-5318_4.patch
>
>
> There are several failing cases when executing the unit tests with Spark 2.2:
> {code}
> org.apache.pig.test.TestAssert#testNegativeWithoutFetch
> org.apache.pig.test.TestAssert#testNegative
> org.apache.pig.test.TestEvalPipeline2#testNonStandardDataWithoutFetch
> org.apache.pig.test.TestScalarAliases#testScalarErrMultipleRowsInInput
> org.apache.pig.test.TestStore#testCleanupOnFailureMultiStore
> org.apache.pig.test.TestStoreInstances#testBackendStoreCommunication
> org.apache.pig.test.TestStoreLocal#testCleanupOnFailureMultiStore
> {code}
> All of these are related to fixes/changes in Spark.
> TestAssert, TestScalarAliases and TestEvalPipeline2 failures could be fixed
> by asserting on the message of the exception's root cause, looks like on
> Spark 2.2 the exception is wrapped into an additional layer.
> TestStore and TestStoreLocal failure are also a test related problems: looks
> like SPARK-7953 is fixed in Spark 2.2
> The root cause of TestStoreInstances is yet to be found out.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
