Re: Unit test best practice for Spark-derived projects

2014-08-07 Thread Patrick Wendell
ookup >> most >> likely), it can take up 30 seconds to start up locally. I think a hosts >> file >> entry is sufficient. >> >> >> >> - >> -- >> Madhu >> https://www.linkedin.com/in/msiddalingaiah >> -- >> View this me

Re: Unit test best practice for Spark-derived projects

2014-08-07 Thread Dmitriy Lyubimov
spark-developers-list.1001551.n3.nabble.com/Unit-test-best-practice-for-Spark-derived-projects-tp7704p7731.html > Sent from the Apache Spark Developers List mailing list archive at > Nabble.com. > > - > To unsubscribe, e

Re: Unit test best practice for Spark-derived projects

2014-08-07 Thread Madhu
this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Unit-test-best-practice-for-Spark-derived-projects-tp7704p7731.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. ---

Unit test best practice for Spark-derived projects

2014-08-05 Thread Dmitriy Lyubimov
Hello, I 've been switching Mahout from Spark 0.9 to Spark 1.0.x [1] and noticed that tests now run much slower compared to 0.9 with CPU running idle most of the time. I had to conclude that most of that time is spent on tearing down/resetting Spark context which apparently now takes significantly