Hi all,
is Mahout 0.13.0 supposed to work with Spark 1.6.3? I would think so as the
master-pom.xml explicitly references Spark 1.6.3.
But when I run a spark-itemsimilarity command (on the 0.13.0-RC) against my
Spark 1.6.3-standalone cluster, the command
fails with:
17/03/03 10:08:40 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0,
reco-master): java.io.IOException: org.apache.spark.SparkException: Failed to
register classes with Kryo
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1212)
at
org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
...
Caused by: java.lang.ClassNotFoundException:
org.apache.mahout.sparkbindings.io.MahoutKryoRegistrator
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
at
org.apache.spark.serializer.KryoSerializer$$anonfun$newKryo$5.apply(KryoSerializer.scala:123)
at scala.Option.map(Option.scala:145)
at
org.apache.spark.serializer.KryoSerializer.newKryo(KryoSerializer.scala:123)
When I run the exactly same command on the 0.12.2 release distribution against
the same Spark cluster, the
command completes sucessfully.
My Environment is:
* Ubuntu 14.04
* Oracle-JDK 1.8.0_121
* Spark standalone cluster using this distribution:
http://d3kbcqa49mib13.cloudfront.net/spark-1.6.3-bin-hadoop2.6.tgz
* Mahout 0.13.0-RC:
https://repository.apache.org/content/repositories/orgapachemahout-1034/org/apache/mahout/apache-mahout-distribution/0.13.0/apache-mahout-distribution-0.13.0.tar.gz
TIA
--
Michael Müller
Condat AG, Berlin
