Sonal's suggestion of looking at the JavaAPISuite is a good idea. Just a few things to note.
Pay special attention to what's being done in the setUp and tearDown methods, because that's where the magic is happening. To unit test against Spark, pretty much all you need to do is create a context running against the "local" master (or "local[*]", etc.). But note that the context is stopped in the tearDown method - this is critical, because if you have multiple local contexts running you can get some very odd errors. If your tests run in parallel, you'll need to share a context instead (via some sort of shared reference). It isn't strictly necessary (I think) to stop the context in this case, since it will be killed when the JVM shuts down. In your tests you can create input data using SparkContext.parallelize and retrieve the output data using RDD.collect (or other actions). On Tue, Jul 29, 2014 at 12:57 PM, Sonal Goyal <sonalgoy...@gmail.com> wrote: > You can take a look at > https://github.com/apache/spark/blob/master/core/src/test/java/org/apache/spark/JavaAPISuite.java > and model your junits based on it. > > Best Regards, > Sonal > Nube Technologies <http://www.nubetech.co> > > <http://in.linkedin.com/in/sonalgoyal> > > > > > On Tue, Jul 29, 2014 at 10:10 PM, Kostiantyn Kudriavtsev < > kudryavtsev.konstan...@gmail.com> wrote: > >> Hi, >> >> try this one >> http://simpletoad.blogspot.com/2014/07/runing-spark-unit-test-on-windows-7.html >> >> it’s more about fixing windows-specific issue, but code snippet gives >> general idea >> just run etl and check output w/ Assert(s) >> >> On Jul 29, 2014, at 6:29 PM, soumick86 <sdasgu...@dstsystems.com> wrote: >> >> > Is there any example out there for unit testing a Spark application in >> Java? >> > Even a trivial application like word count will be very helpful. I am >> very >> > new to this and I am struggling to understand how I can use JavaSpark >> > Context for JUnit >> > >> > >> > >> > -- >> > View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Unit-Testing-JUnit-with-Spark-tp10861.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> > -- Daniel Siegmann, Software Developer Velos Accelerating Machine Learning 440 NINTH AVENUE, 11TH FLOOR, NEW YORK, NY 10001 E: daniel.siegm...@velos.io W: www.velos.io
