Just wanting to +1 this idea. One potential option would be to look at migrating away from the AMP Lab Jenkins infra into the ASF infra. I've added Josh, Shane, and Sean to the CC line explicitly since I think they might have opinions about this.
On Tue, Oct 31, 2017 at 11:05 PM, Xin Lu <x...@salesforce.com> wrote: > Hi everyone, > > I tried sending emails to this list and I'm not sure if it went through so > I'm trying again. Anyway, a couple months ago before I left Databricks I > was working on a proof of concept that parallelized Spark tests on > jenkins. The way it worked was basically it build the spark jars and then > ran all the tests in a docker container on a bunch of slaves in parallel. > This cut the testing time down from 4 hours to approximately 1.5 hours. > This required a newer version of jenkins and the Jenkins Pipeline plugin. > I am wondering if it is possible to do this on amplab jenkins. It looks > like https://builds.apache.org/ has upgraded so Amplabs jenkins is a year > or so behind. I am happy to help with this project if it is something that > people think is worthwhile. > > Thanks > > Xin > -- Twitter: https://twitter.com/holdenkarau
