Re: Moving PR Builder to mvn

2014-10-24 Thread Mark Hamstra
Your's are in the same ballpark with mine, where maven builds with zinc take about 1.4x the time to build with SBT. On Fri, Oct 24, 2014 at 4:24 PM, Sean Owen wrote: > Here's a crude benchmark on a Linux box (GCE n1-standard-4). zinc gets > the assembly build in range of SBT's time. > > mvn -Dsk

Re: Moving PR Builder to mvn

2014-10-24 Thread Sean Owen
Here's a crude benchmark on a Linux box (GCE n1-standard-4). zinc gets the assembly build in range of SBT's time. mvn -DskipTests clean package 15:27 (start zinc) 8:18 (rebuild) 7:08 ./sbt/sbt -DskipTests clean assembly 5:10 (start zinc) 5:11 (rebuild) 5:06 The dependencies were already download

Re: Moving PR Builder to mvn

2014-10-24 Thread Stephen Boesch
Zinc absolutely helps - feels like makes builds more than twice as fast - both on Mac and Linux. It helps both on fresh and existing builds. 2014-10-24 14:06 GMT-07:00 Patrick Wendell : > Does Zinc still help if you are just running a single totally fresh > build? For the pull request builder

Re: Moving PR Builder to mvn

2014-10-24 Thread Hari Shreedharan
+1. From what I can see, it definitely does - though I must say I rarely do full end to end builds though. Maybe worth running as an experiment? Thanks, Hari On Fri, Oct 24, 2014 at 2:34 PM, Stephen Boesch wrote: > Zinc absolutely helps - feels like makes builds more than twice as fast - > b

Re: Moving PR Builder to mvn

2014-10-24 Thread Patrick Wendell
Does Zinc still help if you are just running a single totally fresh build? For the pull request builder we purge all state from previous builds. - Patrick On Fri, Oct 24, 2014 at 1:55 PM, Hari Shreedharan wrote: > I have zinc server running on my mac, and I see maven compilation to be much > bet

Re: Moving PR Builder to mvn

2014-10-24 Thread Hari Shreedharan
I have zinc server running on my mac, and I see maven compilation to be much better than before I had it running. Is the sbt build still faster (sorry, long time since I did a build with sbt). Thanks, Hari On Fri, Oct 24, 2014 at 1:46 PM, Patrick Wendell wrote: > Overall I think this would b

Re: Moving PR Builder to mvn

2014-10-24 Thread Patrick Wendell
Overall I think this would be a good idea. The main blocker is just that I think the Maven build is much slower right now than the SBT build. However, if we were able to e.g. parallelize the test build on Jenkins that might make up for it. I'd actually like to have a trigger where we could tests p

Moving PR Builder to mvn

2014-10-24 Thread Hari Shreedharan
Over the last few months, it seems like we have selected Maven to be the “official” build system for Spark.  I realize that removing the sbt build may not be easy, but it might be a good idea to start looking into that. We had issues over the past few days where mvn builds were fine, while sbt