Hi all,
I've just launched a new Amazon EMR cluster and used the script at:
s3://support.elasticmapreduce/spark/install-spark
to install Spark (this script was upgraded to support 1.2).
I know there are tools to launch a Spark cluster in EC2, but I want to use
EMR.
Everything installs fine; however, when I go to read from a Parquet file, I
end up with (the main exception):
Caused by: java.lang.NoSuchMethodError:
parquet.hadoop.ParquetInputSplit.<init>(Lorg/apache/hadoop/fs/Path;JJJ[Ljava/lang/String;[JLjava/lang/String;Ljava/util/Map;)V
at
parquet.hadoop.TaskSideMetadataSplitStrategy.generateTaskSideMDSplits(ParquetInputFormat.java:578)
... 55 more
It seems to me like a version mismatch somewhere. Where is the
parquet-hadoop jar coming from? Is it built into a fat jar for Spark?
Any help would be appreciated. Note that 1.1.1 worked fine with Parquet
files.
