The problem was mine: I was using FunSuite in the wrong way. The test
method of FunSuite registers a "test" to be triggered when the tests are
running. My localTest is instead creating and stopping the SparkContext
during the test registration and as result my SparkContext is stopped
when the tests are run. The way to fix this is to wrap everything inside
the test method:
import org.scalatest.FunSuite
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
class SparkFunSuite extends FunSuite {
def localTest(name : String)(f : SparkContext => Unit) : Unit = {
this.test(name) {
val conf = new SparkConf()
.setAppName(name)
.setMaster("local")
.set("spark.default.parallelism", "1")
val sc = new SparkContext(conf)
try {
f(sc)
} finally {
sc.stop()
}
}
}
}
class MyTestSuite extends SparkFunSuite {
localTest("My Spark test") { sc =>
assertResult(2)(sc.parallelize(Seq(1,2,3)).filter(_ <= 2).map(_ +
1).count)
}
}
Thanks anyway
On 06.10.2014 20:13, Matei Zaharia wrote:
Weird, it seems like this is trying to use the SparkContext before
it's initialized, or something like that. Have you tried unrolling
this into a single method? I wonder if you just have multiple versions
of these libraries on your classpath or something.
Matei
On Oct 4, 2014, at 1:40 PM, Mario Pastorelli
<mario.pastore...@teralytics.ch
<mailto:mario.pastore...@teralytics.ch>> wrote:
I would like to use FunSuite
<http://doc.scalatest.org/2.2.1/index.html#org.scalatest.FunSuite> to
test my Spark jobs by extending FunSuite with a new function, called
|localTest|, that runs a test with a default SparkContext:
|class SparkFunSuite extends FunSuite {
def localTest(name: String)(f: SparkContext => Unit) : Unit = {
val conf= new SparkConf().setAppName(name).setMaster("local")
val sc= new SparkContext(conf)
try {
this.test(name)(f(sc))
} finally {
sc.stop
}
}
}|
Then I can add tests easily to my testing suites:
|class MyTestSuite extends SparkFunSuite {
localTest("My Spark test") { sc=>
assertResult(2)(sc.parallelize(Seq(1,2,3)).filter(_<= 2).map(_+ 1).count)
}
}|
The problem is that when I run the tests I get a |NullPointerException|:
|[info] MyTestSuite:
[info] - My Spark test*** FAILED***
[info] java.lang.NullPointerException:
[info] at
org.apache.spark.SparkContext.defaultParallelism(SparkContext.scala:1215)
[info] at
org.apache.spark.SparkContext.parallelize$default$2(SparkContext.scala:435)
[info] atMyTestSuite$$anonfun$1.apply(FunSuiteTest.scala:24)
[info] atMyTestSuite$$anonfun$1.apply(FunSuiteTest.scala:23)
[info]
atSparkFunSuite$$anonfun$localTest$1.apply$mcV$sp(FunSuiteTest.scala:13)
[info] atSparkFunSuite$$anonfun$localTest$1.apply(FunSuiteTest.scala:13)
[info] atSparkFunSuite$$anonfun$localTest$1.apply(FunSuiteTest.scala:13)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
[info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info] ...|
What is causing the |NullPointerException|? More importantly, why I
get this error while the testing suite of Spark, that does something
similar, doesn't? Can somebody explain me the difference between them?
I'm using Scala 2.10.4 with |spark-core| 1.0.2 and |scalatest| 2.2.2.
