posted a JIRA https://issues.apache.org/jira/browse/SPARK-1952
On Wed, May 28, 2014 at 1:14 PM, Ryan Compton wrote:
> Remark, just including the jar built by sbt will produce the same
> error. i,.e this pig script will fail:
>
> REGISTER
> /usr/share/osi1/spark-1.0.0/assembly/target/scala-2.10/s
Remark, just including the jar built by sbt will produce the same
error. i,.e this pig script will fail:
REGISTER
/usr/share/osi1/spark-1.0.0/assembly/target/scala-2.10/spark-assembly-1.0.0-SNAPSHOT-hadoop0.20.2-cdh3u4.jar;
edgeList0 = LOAD
'/user/rfcompton/twitter-mention-networks/bidirectional
It appears to be Spark 1.0 related. I made a pom.xml with a single
dependency on Spark, registering the resulting jar created the error.
Spark 1.0 was compiled via $ SPARK_HADOOP_VERSION=0.20.2-cdh3u4 sbt/sbt assembly
The pom.xml, as well as some other information, is below. The only
thing that s
Spark uses 1.7.5, and you should probably see 1.7.{4,5} in use through
Hadoop. But those are compatible.
That method appears to have been around since 1.3. What version does Pig want?
I usually do "mvn -Dverbose dependency:tree" to see both what the
final dependencies are, and what got overwritte
