Re: Unit tests can generate spurious shutdown messages

Can you submit a pull request for it? Thanks. On Tue, Jun 2, 2015 at 4:25 AM, Mick Davies wrote: > If I write unit tests that indirectly initialize > org.apache.spark.util.Utils, > for example use sql types, but produce no logging, I get the following > unpleasant stack trace in my test output.

Re: Unit tests

Thank, Josh, I missed that PR. On Mon, Feb 9, 2015 at 7:45 PM, Josh Rosen wrote: > Hi Iulian, > > I think the AkakUtilsSuite failure that you observed has been fixed in > https://issues.apache.org/jira/browse/SPARK-5548 / > https://github.com/apache/spark/pull/4343 > > On February 9, 2015 at 5:4

Re: Unit tests

Hi Iulian, I think the AkakUtilsSuite failure that you observed has been fixed in  https://issues.apache.org/jira/browse/SPARK-5548 /  https://github.com/apache/spark/pull/4343 On February 9, 2015 at 5:47:59 AM, Iulian Dragoș (iulian.dra...@typesafe.com) wrote: Hi Patrick, Thanks for the head

Re: Unit tests

Hi Patrick, Thanks for the heads up. I was trying to set up our own infrastructure for testing Spark (essentially, running `run-tests` every night) on EC2. I stumbled upon a number of flaky tests, but none of them look similar to anything in Jira with the flaky-test tag. I wonder if there's someth

Re: Unit tests in < 5 minutes

bq. I may move on to trying Maven. Maven is my favorite :-) On Sat, Dec 6, 2014 at 10:54 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Ted, > > I posted some updates >

Re: Unit tests in < 5 minutes

Ted, I posted some updates on JIRA on my progress (or lack thereof) getting SBT to parallelize test suites properly. I'm currently stu

Re: Unit tests in < 5 minutes

@Patrick and Josh actually we went even further than that. We simply disable the UI for most tests and these used to be the single largest source of port conflict.

Re: Unit tests in < 5 minutes

Have you seen this thread http://search-hadoop.com/m/JW1q5xxSAa2 ? Test categorization in HBase is done through maven-surefire-plugin Cheers On Thu, Dec 4, 2014 at 4:05 PM, Nicholas Chammas wrote: > fwiw, when we did this work in HBase, we categorized the tests. Then some > tests can share a s

Re: Unit tests in < 5 minutes

fwiw, when we did this work in HBase, we categorized the tests. Then some tests can share a single jvm, while some others need to be isolated in their own jvm. Nevertheless surefire can still run them in parallel by starting/stopping several jvm. I think we need to do this as well. Perhaps the tes

Re: Unit tests in < 5 minutes

On Fri, Aug 8, 2014 at 1:12 PM, Reynold Xin wrote: > Nick, > > Would you like to file a ticket to track this? > SPARK-3431 : Parallelize execution of tests > Sub-task: SPARK-3432 : Fix logging of

Re: Unit tests in < 5 minutes

Issue with supporting this imo is the fact that scala-test uses the same vm for all the tests (surefire plugin supports fork, but scala-test ignores it iirc). So different tests would initialize different spark context, and can potentially step on each others toes. Regards, Mridul On Fri, Aug 8,

Re: Unit tests in < 5 minutes

Josh - that was actually fixed recently (we just bind to a random port when running tests). On Fri, Aug 8, 2014 at 12:00 PM, Josh Rosen wrote: > One simple optimization might be to disable the application web UI in tests > that don't need it. When running tests on my local machine while also > r

Re: Unit tests in < 5 minutes

One simple optimization might be to disable the application web UI in tests that don’t need it.  When running tests on my local machine while also running another Spark shell, I’ve noticed that the test logs fill up with errors when the web UI attempts to bind to the default port, fails, and tri

Re: Unit tests in < 5 minutes

I dug around this a bit a while ago, I think if someone sat down and profiled the tests it's likely we could find some things to optimize. In particular, there may be overheads in starting up a local spark context that could be minimized and speed up all the tests. Also, there are some tests (espec

Re: Unit tests in < 5 minutes

Just as a note, when you're developing stuff, you can use "test-only" in sbt, or the equivalent feature in Maven, to run just some of the tests. This is what I do, I don't wait for Jenkins to run things. 90% of the time if it passes the tests that I know could break stuff, it will pass all of Je

Re: Unit tests in < 5 minutes

fwiw, when we did this work in HBase, we categorized the tests. Then some tests can share a single jvm, while some others need to be isolated in their own jvm. Nevertheless surefire can still run them in parallel by starting/stopping several jvm. Nicolas On Fri, Aug 8, 2014 at 7:10 PM, Reynold X

Re: Unit tests in < 5 minutes

Nick, Would you like to file a ticket to track this? I think the first baby step is to log the amount of time each test cases take. This is supposed to happen already (see the flag), but somehow the time are not showing. If you have some time to figure that out, that'd be great. https://github.c

Re: Unit tests in < 5 minutes

ScalaTest actually has support for parallelization built-in. We can use that. The main challenge is to make sure all the test suites can work in parallel when running along side each other. On Fri, Aug 8, 2014 at 9:47 AM, Ted Yu wrote: > How about using parallel execution feature of maven-sure

Re: Unit tests in < 5 minutes

How about using parallel execution feature of maven-surefire-plugin (assuming all the tests were made parallel friendly) ? http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html Cheers On Fri, Aug 8, 2014 at 9:14 AM, Sean Owen wrote: > A commo

Re: Unit tests in < 5 minutes

A common approach is to separate unit tests from integration tests. Maven has support for this distinction. I'm not sure it helps a lot though, since it only helps you to not run integration tests all the time. But lots of Spark tests are integration-test-like and are important to run to know a cha