We actually have some immediate needs for custom config for some upcoming 
integration tests.

I don't know if such changes are possible in ASF Jenkins but the work is in 
progress in RISELab Jenkins :)


________________________________
From: holden.ka...@gmail.com <holden.ka...@gmail.com> on behalf of Holden Karau 
<hol...@pigscanfly.ca>
Sent: Tuesday, November 7, 2017 2:14:18 PM
To: Sean Owen
Cc: Xin Lu; dev@spark.apache.org
Subject: Re: Jenkins upgrade/Test Parallelization & Containerization

True, I think we've seen that the Amp Lab Jenkins needs to be more focused on 
running AMP Lab projects, and while I don't know how difficult the ASF Jenkins 
is I assume it might be an easier place to make changes going forward? (Of 
course this could be the grass is greener on the other side and I don't mean to 
say it's been hard to make changes on the AMP lab hardware, folks have been 
amazingly helpful - its just the projects on each have different needs).

On Tue, Nov 7, 2017 at 12:52 PM, Sean Owen 
<so...@cloudera.com<mailto:so...@cloudera.com>> wrote:
Faster tests would be great. I recall that the straightforward ways to 
parallelize via Maven haven't worked because many tests collide with one 
another. Is this about running each module's tests in a container? that should 
work.

I can see how this is becoming essential for repeatable and reliable Python/R 
builds, which depend on the environment to a much greater extent than the JVM 
does.

I don't have a strong preference for AMPLab vs ASF builds. I suppose using the 
ASF machinery is a little tidier. If it's got a later Jenkins that's required, 
also a plus, but I assume updating AMPLab isn't so hard here either. I think 
the key issue is which environment is easier to control and customize over time.


On Wed, Nov 1, 2017 at 6:05 AM Xin Lu 
<x...@salesforce.com<mailto:x...@salesforce.com>> wrote:
Hi everyone,

I tried sending emails to this list and I'm not sure if it went through so I'm 
trying again.  Anyway, a couple months ago before I left Databricks I was 
working on a proof of concept that parallelized Spark tests on jenkins.  The 
way it worked was basically it build the spark jars and then ran all the tests 
in a docker container on a bunch of slaves in parallel.  This cut the testing 
time down from 4 hours to approximately 1.5 hours.  This required a newer 
version of jenkins and the Jenkins Pipeline plugin.  I am wondering if it is 
possible to do this on amplab jenkins.  It looks like 
https://builds.apache.org/ has upgraded so Amplabs jenkins is a year or so 
behind.  I am happy to help with this project if it is something that people 
think is worthwhile.

Thanks

Xin



--
Twitter: https://twitter.com/holdenkarau

Reply via email to