That makes sense, in that case do we know how hard it would be to make the necessary hands to the AMP Lab Jenkins to support this?
On Tue, Nov 7, 2017 at 4:04 PM Reynold Xin <r...@databricks.com> wrote: > My understanding is that AMP actually can provide more resources or adapt > changes, while ASF needs to manage 200+ projects and it's hard to > accommodate much. I could be wrong though. > > > On Tue, Nov 7, 2017 at 2:14 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > >> True, I think we've seen that the Amp Lab Jenkins needs to be more >> focused on running AMP Lab projects, and while I don't know how difficult >> the ASF Jenkins is I assume it might be an easier place to make changes >> going forward? (Of course this could be the grass is greener on the other >> side and I don't mean to say it's been hard to make changes on the AMP lab >> hardware, folks have been amazingly helpful - its just the projects on each >> have different needs). >> >> On Tue, Nov 7, 2017 at 12:52 PM, Sean Owen <so...@cloudera.com> wrote: >> >>> Faster tests would be great. I recall that the straightforward ways to >>> parallelize via Maven haven't worked because many tests collide with one >>> another. Is this about running each module's tests in a container? that >>> should work. >>> >>> I can see how this is becoming essential for repeatable and reliable >>> Python/R builds, which depend on the environment to a much greater extent >>> than the JVM does. >>> >>> I don't have a strong preference for AMPLab vs ASF builds. I suppose >>> using the ASF machinery is a little tidier. If it's got a later Jenkins >>> that's required, also a plus, but I assume updating AMPLab isn't so hard >>> here either. I think the key issue is which environment is easier to >>> control and customize over time. >>> >>> >>> On Wed, Nov 1, 2017 at 6:05 AM Xin Lu <x...@salesforce.com> wrote: >>> >>>> Hi everyone, >>>> >>>> I tried sending emails to this list and I'm not sure if it went through >>>> so I'm trying again. Anyway, a couple months ago before I left Databricks >>>> I was working on a proof of concept that parallelized Spark tests on >>>> jenkins. The way it worked was basically it build the spark jars and then >>>> ran all the tests in a docker container on a bunch of slaves in parallel. >>>> This cut the testing time down from 4 hours to approximately 1.5 hours. >>>> This required a newer version of jenkins and the Jenkins Pipeline plugin. >>>> I am wondering if it is possible to do this on amplab jenkins. It looks >>>> like https://builds.apache.org/ has upgraded so Amplabs jenkins is a >>>> year or so behind. I am happy to help with this project if it is something >>>> that people think is worthwhile. >>>> >>>> Thanks >>>> >>>> Xin >>>> >>> >> >> >> -- >> Twitter: https://twitter.com/holdenkarau >> > > -- Twitter: https://twitter.com/holdenkarau
