Re: Unit Testing (JUnit) with Spark

add these to your dependencies: "io.netty" % "netty" % "3.6.6.Final" exclude("io.netty", "netty-all") to the end of spark and hadoop dependencies reference: https://spark-project.atlassian.net/browse/SPARK-1138 I am using Spark 1.1 so the akka issue is already fixed -- View this message in co

Re: Unit Testing (JUnit) with Spark

Few lines of my error logs look like 2014-07-29 16:32:16,326 ERROR [ActorSystemImpl] Uncaught fatal error from thread [spark-akka.actor.default-dispatcher-6] shutting down ActorSystem [spark] java.lang.VerifyError: (class: org/jboss/netty/channel/socket/nio/NioWorkerPool, method: createWorker si

Re: Unit Testing (JUnit) with Spark

Sonal's suggestion of looking at the JavaAPISuite is a good idea. Just a few things to note. Pay special attention to what's being done in the setUp and tearDown methods, because that's where the magic is happening. To unit test against Spark, pretty much all you need to do is create a context run

Re: Unit Testing (JUnit) with Spark

You can take a look at https://github.com/apache/spark/blob/master/core/src/test/java/org/apache/spark/JavaAPISuite.java and model your junits based on it. Best Regards, Sonal Nube Technologies On Tue, Jul 29, 2014 at 10:10 PM, K

Re: Unit Testing (JUnit) with Spark

Hi, try this one http://simpletoad.blogspot.com/2014/07/runing-spark-unit-test-on-windows-7.html it’s more about fixing windows-specific issue, but code snippet gives general idea just run etl and check output w/ Assert(s) On Jul 29, 2014, at 6:29 PM, soumick86 wrote: > Is there any example

Re: Unit Testing (JUnit) with Spark

I've been working some on building spark blueprints, and recently tried to generalize one for easy blueprints of spark apps. https://github.com/jayunit100/SparkBlueprint.git It runs the spark app's main method in a unit test, and builds in SBT. You can easily try it out and improve on it. Obvio