spark-itemsimilarity: Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.event-handlers'
Hi, I'm new to Spark, Mahout. Just tried to run the spark-itemsimilarity but no luck. Here is what I did. 1. clone git Mahout git project 2. run "mvn install" 3. set $MAHOUT_HOME to the project path 4. added $MAHOUT_HOME/bin to the PATH variable 5. download Spark, extract it to a dir 6. set $SPARK_HOME to the dir /spark-1.2.0-bin-hadoop2.4 7. run command "mahout spark-itemsimilarity -i ./mahout-input/order_item.tsv -o ./output -f1 purchase -f2 view -os -ic 2 -fc 1 -td ," Below is the output. It says "Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.event-handlers'" Any help or suggestion is highly appreciated. Thanks, Kevin -- MAHOUT_LOCAL is set, so we don't add HADOOP_CONF_DIR to classpath. log4j:WARN No appenders could be found for logger (org.apache.mahout.sparkbindings). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/Zhang_Kevin/Documents/mine/big/projects/mahout/mrlegacy/target/mahout-mrlegacy-1.0-SNAPSHOT-job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/Zhang_Kevin/Documents/mine/big/projects/mahout/spark/target/mahout-spark_2.10-1.0-SNAPSHOT-job.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/Zhang_Kevin/Applications/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 15/01/27 12:01:58 INFO SecurityManager: Changing view acls to: Zhang_Kevin, 15/01/27 12:01:58 INFO SecurityManager: Changing modify acls to: Zhang_Kevin, 15/01/27 12:01:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(Zhang_Kevin, ); users with modify permissions: Set(Zhang_Kevin, ) Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.event-handlers' at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115) at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136) at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142) at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150) at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155) at com.typesafe.config.impl.SimpleConfig.getList(SimpleConfig.java:203) at com.typesafe.config.impl.SimpleConfig.getHomogeneousUnwrappedList(SimpleConfig.java:260) at com.typesafe.config.impl.SimpleConfig.getStringList(SimpleConfig.java:318) at akka.actor.ActorSystem$Settings.(ActorSystem.scala:150) at akka.actor.ActorSystemImpl.(ActorSystem.scala:470) at akka.actor.ActorSystem$.apply(ActorSystem.scala:111) at akka.actor.ActorSystem$.apply(ActorSystem.scala:104) at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446) at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442) at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) at org.apache.spark.SparkEnv$.create(SparkEnv.scala:153) at org.apache.spark.SparkContext.(SparkContext.scala:203) at org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:95) at org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:81) at org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:128) at org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:211) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:116) at org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:114) at scala.Option.map(Option.scala:145) at org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDriver.scala:114) at org.apache.mahout.drivers.ItemSimilarityDriver.main(ItemSimilarityDriver.scala)
Re: spark-itemsimilarity: Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.event-handlers'
spark 1.2 not supported (yet). current head runs on 1.1.0 (but i guess you can take a pull request #71 and compile it for 1.1.1 too, and perhaps even 1.2) On Tue, Jan 27, 2015 at 12:04 PM, Kevin Zhang < zhangyongji...@yahoo.com.invalid> wrote: > Hi, > > I'm new to Spark, Mahout. Just tried to run the spark-itemsimilarity but > no luck. > > Here is what I did. > > 1. clone git Mahout git project > 2. run "mvn install" > 3. set $MAHOUT_HOME to the project path > 4. added $MAHOUT_HOME/bin to the PATH variable > 5. download Spark, extract it to a dir > 6. set $SPARK_HOME to the dir /spark-1.2.0-bin-hadoop2.4 > 7. run command "mahout spark-itemsimilarity -i > ./mahout-input/order_item.tsv -o ./output -f1 purchase -f2 view -os -ic 2 > -fc 1 -td ," > > Below is the output. It says "Exception in thread "main" > com.typesafe.config.ConfigException$Missing: No configuration setting found > for key 'akka.event-handlers'" > > Any help or suggestion is highly appreciated. > > Thanks, > Kevin > > > -- > MAHOUT_LOCAL is set, so we don't add HADOOP_CONF_DIR to classpath. > log4j:WARN No appenders could be found for logger > (org.apache.mahout.sparkbindings). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for > more info. > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/Users/Zhang_Kevin/Documents/mine/big/projects/mahout/mrlegacy/target/mahout-mrlegacy-1.0-SNAPSHOT-job.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/Zhang_Kevin/Documents/mine/big/projects/mahout/spark/target/mahout-spark_2.10-1.0-SNAPSHOT-job.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/Users/Zhang_Kevin/Applications/spark-1.2.0-bin-hadoop2.4/lib/spark-assembly-1.2.0-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > 15/01/27 12:01:58 INFO SecurityManager: Changing view acls to: Zhang_Kevin, > 15/01/27 12:01:58 INFO SecurityManager: Changing modify acls to: > Zhang_Kevin, > 15/01/27 12:01:58 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(Zhang_Kevin, > ); users with modify permissions: Set(Zhang_Kevin, ) > Exception in thread "main" com.typesafe.config.ConfigException$Missing: No > configuration setting found for key 'akka.event-handlers' > at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:115) > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136) > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142) > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150) > at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155) > at com.typesafe.config.impl.SimpleConfig.getList(SimpleConfig.java:203) > at > com.typesafe.config.impl.SimpleConfig.getHomogeneousUnwrappedList(SimpleConfig.java:260) > at > com.typesafe.config.impl.SimpleConfig.getStringList(SimpleConfig.java:318) > at akka.actor.ActorSystem$Settings.(ActorSystem.scala:150) > at akka.actor.ActorSystemImpl.(ActorSystem.scala:470) > at akka.actor.ActorSystem$.apply(ActorSystem.scala:111) > at akka.actor.ActorSystem$.apply(ActorSystem.scala:104) > at > org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121) > at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) > at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) > at > org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1446) > at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141) > at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1442) > at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) > at org.apache.spark.SparkEnv$.create(SparkEnv.scala:153) > at org.apache.spark.SparkContext.(SparkContext.scala:203) > at > org.apache.mahout.sparkbindings.package$.mahoutSparkContext(package.scala:95) > at > org.apache.mahout.drivers.MahoutSparkDriver.start(MahoutSparkDriver.scala:81) > at > org.apache.mahout.drivers.ItemSimilarityDriver$.start(ItemSimilarityDriver.scala:128) > at > org.apache.mahout.drivers.ItemSimilarityDriver$.process(ItemSimilarityDriver.scala:211) > at > org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:116) > at > org.apache.mahout.drivers.ItemSimilarityDriver$$anonfun$main$1.apply(ItemSimilarityDriver.scala:114) > at scala.Option.map(Option.scala:145) > at > org.apache.mahout.drivers.ItemSimilarityDriver$.main(ItemSimilarityDri
java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
Thanks to Dmitriy for answering my previous question regarding the Spark version. I just downgraded the version to spark-1.1.0-bin-hadoop2.4 and run my commane "mahout spark-itemsimilarity -i ./mahout-input/order_item.tsv -o ./output -f1 purchase -f2 view -os -ic 2 -fc 1 -td ," again. This time I got error "java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path" as attached. I'm using Mac. Thanks for the help -Kevin at org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.scala:65) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) at org.apache.spark.scheduler.Task.run(Task.scala:54) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886) at java.lang.Runtime.loadLibrary0(Runtime.java:849) at java.lang.System.loadLibrary(System.java:1088) at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52) ... 26 more 15/01/27 14:54:24 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 0) org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) at org.xerial.snappy.Snappy.(Snappy.java:44) at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:79) at org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125) at org.apache.spark.storage.BlockManager.wrapForCompression(BlockManager.scala:1029) at org.apache.spark.storage.BlockManager$$anonfun$8.apply(BlockManager.scala:608) at org.apache.spark.storage.BlockManager$$anonfun$8.apply(BlockManager.scala:608)
Re: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
This looks like hadoop or spark -specific thing (snappy codec is used by spark by default). There should be a way to disable this to a more palatable library but you will need to investigate it a little bit since i don't think anybody here knows mac specifics. Better yet is to figure how to install native snappy codec on your Mac. There should be a way. Ask on spark list. To switch codec to something else you may try to get something like this into system properties of the driver process: -Dspark.io.compression.codec=lzf Now, normally it would be as easy as adding it to MAHOUT_OPTS environment; however, i think our current head is broken w.r.t. MAHOUT_OPTS for spark processes (i have a fix for it elsewhere but not in public branch). So if you decide to switch codec, you may need hack bin/mahout script a little, not sure. On Tue, Jan 27, 2015 at 3:07 PM, Kevin Zhang < zhangyongji...@yahoo.com.invalid> wrote: > Thanks to Dmitriy for answering my previous question regarding the Spark > version. I just downgraded the version to spark-1.1.0-bin-hadoop2.4 and run > my commane "mahout spark-itemsimilarity -i ./mahout-input/order_item.tsv -o > ./output -f1 purchase -f2 view -os -ic 2 -fc 1 -td ," again. This time I > got error "java.lang.UnsatisfiedLinkError: no snappyjava in > java.library.path" as attached. I'm using Mac. > > Thanks for the help > -Kevin > > > at > org.apache.spark.shuffle.hash.HashShuffleWriter.write(HashShuffleWriter.scala:65) > > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41) > at org.apache.spark.scheduler.Task.run(Task.scala:54) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in > java.library.path > at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1886) > at java.lang.Runtime.loadLibrary0(Runtime.java:849) > at java.lang.System.loadLibrary(System.java:1088) > at > org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52) > ... 26 more > 15/01/27 14:54:24 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID > 0) > org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null > at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229) > at org.xerial.snappy.Snappy.(Snappy.java:44) > at org.xerial.snappy.SnappyOutputStream.(SnappyOutputStream.java:79) > at > org.apache.spark.io.SnappyCompressionCodec.compressedOutputStream(CompressionCodec.scala:125) > at > org.apache.spark.storage.BlockManager.wrapForCompression(BlockManager.scala:1029) > at > org.apache.spark.storage.BlockManager$$anonfun$8.apply(BlockManager.scala:608) > at > org.apache.spark.storage.BlockManager$$anonfun$8.apply(BlockManager.scala:608)
