On a related note, I believe Hyukjin has been requesting access to trigger tests on amplab Jenkins for some time. Would be great to add him.
On Wed, Nov 8, 2017, 4:25 AM Felix Cheung <felixcheun...@hotmail.com> wrote: > We actually have some immediate needs for custom config for some upcoming > integration tests. > > I don’t know if such changes are possible in ASF Jenkins but the work is > in progress in RISELab Jenkins :) > > > ------------------------------ > *From:* holden.ka...@gmail.com <holden.ka...@gmail.com> on behalf of > Holden Karau <hol...@pigscanfly.ca> > *Sent:* Tuesday, November 7, 2017 2:14:18 PM > *To:* Sean Owen > *Cc:* Xin Lu; dev@spark.apache.org > *Subject:* Re: Jenkins upgrade/Test Parallelization & Containerization > > True, I think we've seen that the Amp Lab Jenkins needs to be more focused > on running AMP Lab projects, and while I don't know how difficult the ASF > Jenkins is I assume it might be an easier place to make changes going > forward? (Of course this could be the grass is greener on the other side > and I don't mean to say it's been hard to make changes on the AMP lab > hardware, folks have been amazingly helpful - its just the projects on each > have different needs). > > On Tue, Nov 7, 2017 at 12:52 PM, Sean Owen <so...@cloudera.com> wrote: > >> Faster tests would be great. I recall that the straightforward ways to >> parallelize via Maven haven't worked because many tests collide with one >> another. Is this about running each module's tests in a container? that >> should work. >> >> I can see how this is becoming essential for repeatable and reliable >> Python/R builds, which depend on the environment to a much greater extent >> than the JVM does. >> >> I don't have a strong preference for AMPLab vs ASF builds. I suppose >> using the ASF machinery is a little tidier. If it's got a later Jenkins >> that's required, also a plus, but I assume updating AMPLab isn't so hard >> here either. I think the key issue is which environment is easier to >> control and customize over time. >> >> >> On Wed, Nov 1, 2017 at 6:05 AM Xin Lu <x...@salesforce.com> wrote: >> >>> Hi everyone, >>> >>> I tried sending emails to this list and I'm not sure if it went through >>> so I'm trying again. Anyway, a couple months ago before I left Databricks >>> I was working on a proof of concept that parallelized Spark tests on >>> jenkins. The way it worked was basically it build the spark jars and then >>> ran all the tests in a docker container on a bunch of slaves in parallel. >>> This cut the testing time down from 4 hours to approximately 1.5 hours. >>> This required a newer version of jenkins and the Jenkins Pipeline plugin. >>> I am wondering if it is possible to do this on amplab jenkins. It looks >>> like https://builds.apache.org/ has upgraded so Amplabs jenkins is a >>> year or so behind. I am happy to help with this project if it is something >>> that people think is worthwhile. >>> >>> Thanks >>> >>> Xin >>> >> > > > -- > Twitter: https://twitter.com/holdenkarau >
