Re: Spurious test failures, testing best practices

I agree we should separate out the integration tests so it's easy for dev to just run the other fast tests locally. I opened a jira for it https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-4746 On Nov 30, 2014 3:08 PM, "Matei Zaharia" wrote: > Hi Ryan, > > As a tip (and maybe th

Re: Spurious test failures, testing best practices

Thanks Marcelo, "this is just how Maven works (unfortunately)" answers my question. Another related question: I tried to use `mvn scala:cc` and discovered that it only seems to work scan src/main and src/test directories (according to its docs

Re: Spurious test failures, testing best practices

On Tue, Dec 2, 2014 at 4:40 PM, Ryan Williams wrote: >> But you only need to compile the others once. > > once... every time I rebase off master, or am obliged to `mvn clean` by some > other build-correctness bug, as I said before. In my experience this works > out to a few times per week. No, yo

Re: Spurious test failures, testing best practices

On Tue Dec 02 2014 at 4:46:20 PM Marcelo Vanzin wrote: > On Tue, Dec 2, 2014 at 3:39 PM, Ryan Williams > wrote: > > Marcelo: by my count, there are 19 maven modules in the codebase. I am > > typically only concerned with "core" (and therefore its two dependencies > as > > well, `network/{shuffle

Re: Spurious test failures, testing best practices

On Tue, Dec 2, 2014 at 3:39 PM, Ryan Williams wrote: > Marcelo: by my count, there are 19 maven modules in the codebase. I am > typically only concerned with "core" (and therefore its two dependencies as > well, `network/{shuffle,common}`). But you only need to compile the others once. Once you'v

Re: Spurious test failures, testing best practices

Marcelo: by my count, there are 19 maven modules in the codebase. I am typically only concerned with "core" (and therefore its two dependencies as well, `network/{shuffle,common}`). The `mvn package` workflow (and its sbt equivalent) that most people apparently use involves (for me) compiling+pack

Re: Spurious test failures, testing best practices

Hey Ryan, What if you run a single "mvn install" to install all libraries locally - then can you "mvn compile -pl core"? I think this may be the only way to make it work. - Patrick On Tue, Dec 2, 2014 at 2:40 PM, Ryan Williams wrote: > Following on Mark's Maven examples, here is another related

Re: Spurious test failures, testing best practices

On Tue, Dec 2, 2014 at 2:40 PM, Ryan Williams wrote: > Following on Mark's Maven examples, here is another related issue I'm > having: > > I'd like to compile just the `core` module after a `mvn clean`, without > building an assembly JAR first. Is this possible? Out of curiosity, may I ask why? W

Re: Spurious test failures, testing best practices

Following on Mark's Maven examples, here is another related issue I'm having: I'd like to compile just the `core` module after a `mvn clean`, without building an assembly JAR first. Is this possible? Attempting to do it myself, the steps I performed were: - `mvn compile -pl core`: fails because

Re: Spurious test failures, testing best practices

Hi Ilya - you can just submit a pull request and the way we test them is to run it through jenkins. You don't need to do anything special. On Sun, Nov 30, 2014 at 8:57 PM, Ganelin, Ilya wrote: > Hi, Patrick - with regards to testing on Jenkins, is the process for this > to submit a pull request f

Re: Spurious test failures, testing best practices

Hi, Patrick - with regards to testing on Jenkins, is the process for this to submit a pull request for the branch or is there another interface we can use to submit a build to Jenkins for testing? On 11/30/14, 6:49 PM, "Patrick Wendell" wrote: >Hey Ryan, > >A few more things here. You should fee

Re: Spurious test failures, testing best practices

Thanks Patrick, great to hear that docs-snapshots-via-jenkins is already JIRA'd; you can interpret some of this thread as a gigantic +1 from me on prioritizing that, which it looks like you are doing :) I do understand the limitations of the "github vs. official site" status quo; I was mostly resp

Re: Spurious test failures, testing best practices

Thanks Mark, most of those commands are things I've been using and used in my original post except for "Start zinc". I now see the section about it on the "unpublished" building-spark page and wil

Re: Spurious test failures, testing best practices

Btw - the documnetation on github represents the source code of our docs, which is versioned with each release. Unfortunately github will always try to render ".md" files so it could look to a passerby like this is supposed to represent published docs. This is a feature limitation of github, AFAIK

Re: Spurious test failures, testing best practices

Hey Ryan, The existing JIRA also covers publishing nightly docs: https://issues.apache.org/jira/browse/SPARK-1517 - Patrick On Sun, Nov 30, 2014 at 5:53 PM, Ryan Williams wrote: > Thanks Nicholas, glad to hear that some of this info will be pushed to the > main site soon, but this brings up yet

Re: Spurious test failures, testing best practices

Thanks Nicholas, glad to hear that some of this info will be pushed to the main site soon, but this brings up yet another point of confusion that I've struggled with, namely whether the documentation on github or that on spark.apache.org should be considered the primary reference for people seeking

Re: Spurious test failures, testing best practices

> > - Start the SBT interactive console with sbt/sbt > - Build your assembly by running the "assembly" target in the assembly > project: assembly/assembly > - Run all the tests in one module: core/test > - Run a specific suite: core/test-only org.apache.spark.rdd.RDDSuite (this > also supports tab

Re: Spurious test failures, testing best practices

- currently the docs only contain information about building with maven, and even then don’t cover many important cases All other points aside, I just want to point out that the docs document both how to use Maven and SBT and clearly state

Re: Spurious test failures, testing best practices

Hey Ryan, A few more things here. You should feel free to send patches to Jenkins to test them, since this is the reference environment in which we regularly run tests. This is the normal workflow for most developers and we spend a lot of effort provisioning/maintaining a very large jenkins cluste

Re: Spurious test failures, testing best practices

thanks for the info, Matei and Brennon. I will try to switch my workflow to using sbt. Other potential action items: - currently the docs only contain information about building with maven, and even then don't cover many important cases, as I described in my previous email. If SBT is as much bette

Re: Spurious test failures, testing best practices

Hi Ryan, As a tip (and maybe this isn't documented well), I normally use SBT for development to avoid the slow build process, and use its interactive console to run only specific tests. The nice advantage is that SBT can keep the Scala compiler loaded and JITed across builds, making it faster t

Re: Spurious test failures, testing best practices

+1, you aren¹t alone in this. I certainly would like some clarity in these things well, but, as its been said on this listserv a few times (and you noted), most developers use `sbt` for their day-to-day compilations to greatly speed up the iterative testing process. I personally use `sbt` for all b