[
https://issues.apache.org/jira/browse/PIG-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172741#comment-16172741
]
liyunzhang_intel edited comment on PIG-5305 at 9/20/17 4:47 AM:
----------------------------------------------------------------
{quote}
I also checked, test-tez was not running properly since the Spark 2 support
commit, because setTezEnv was clearing the excluded sources property. I fixed
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in the dependency of
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset
{{src.exclude.dir}} , this will influence {{jar}} which use the property
{{src.exclude.dir}}?
{code}
<target name="jar" description="Create pig jar with Spark 1 and 2">
<echo>Compiling against Spark 2</echo>
<antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
<propertyreset name="sparkversion" value="2"/>
<propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
<antcall target="jar-core" inheritRefs="true" inheritall="true"/>
<move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>
<echo>Compiling against Spark 1</echo>
<antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
<propertyreset name="sparkversion" value="1"/>
<propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
<antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
<jar update="yes" jarfile="${output.jarfile.core}">
<zipfileset src="${basedir}/_pig-shims.jar"
includes="**/Spark2*.class"/>
</jar>
<jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
<zipfileset src="${basedir}/_pig-shims.jar"
includes="**/Spark2*.class"/>
</jar>
<jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
<zipfileset src="${basedir}/_pig-shims.jar"
includes="**/Spark2*.class"/>
</jar>
<delete file="${basedir}/_pig-shims.jar"/>
</target>
{code}
If my understanding is right, why TestEvalPipeline unit test passes in tez mode
before this jira?
was (Author: kellyzly):
{quote}
I also checked, test-tez was not running properly since the Spark 2 support
commit, because setTezEnv was clearing the excluded sources property. I fixed
this in my latest patch as well.
{quote}
what you mean is before we add {{jar-simple}} in the dependency of
{{test-tez}} in PIG-5157, But in {{setTezEnv}}, it will reset
{{src.exclude.dir}} , this will influence {{jar}} which use the property
{{src.exclude.dir}}?
{code}
<target name="jar" description="Create pig jar with Spark 1 and 2">
<echo>Compiling against Spark 2</echo>
<antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
<propertyreset name="sparkversion" value="2"/>
<propertyreset name="src.exclude.dir" value="**/Spark1*.java" />
<antcall target="jar-core" inheritRefs="true" inheritall="true"/>
<move file="${output.jarfile.core}" tofile="${basedir}/_pig-shims.jar"/>
<echo>Compiling against Spark 1</echo>
<antcall target="clean-deps" inheritRefs="true" inheritall="true"/>
<propertyreset name="sparkversion" value="1"/>
<propertyreset name="src.exclude.dir" value="**/Spark2*.java" />
<antcall target="jar-simple" inheritRefs="true" inheritall="true"/>
<jar update="yes" jarfile="${output.jarfile.core}">
<zipfileset src="${basedir}/_pig-shims.jar"
includes="**/Spark2*.class"/>
</jar>
<jar update="yes" jarfile="${output.jarfile.backcompat-core-h2}">
<zipfileset src="${basedir}/_pig-shims.jar"
includes="**/Spark2*.class"/>
</jar>
<jar update="yes" jarfile="${output.jarfile.withouthadoop-h2}">
<zipfileset src="${basedir}/_pig-shims.jar"
includes="**/Spark2*.class"/>
</jar>
<delete file="${basedir}/_pig-shims.jar"/>
</target>
{code}
> Enable yarn-client mode execution of tests in Spark (1) mode
> ------------------------------------------------------------
>
> Key: PIG-5305
> URL: https://issues.apache.org/jira/browse/PIG-5305
> Project: Pig
> Issue Type: Sub-task
> Components: spark
> Reporter: Adam Szita
> Assignee: Adam Szita
> Attachments: PIG-5305.0.patch, PIG-5305.1.patch, PIG-5305.2.patch
>
>
> See parent jira (PIG-5305) for problem description
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
