Hi everyone, I tried sending emails to this list and I'm not sure if it went through so I'm trying again. Anyway, a couple months ago before I left Databricks I was working on a proof of concept that parallelized Spark tests on jenkins. The way it worked was basically it build the spark jars and then ran all the tests in a docker container on a bunch of slaves in parallel. This cut the testing time down from 4 hours to approximately 1.5 hours. This required a newer version of jenkins and the Jenkins Pipeline plugin. I am wondering if it is possible to do this on amplab jenkins. It looks like https://builds.apache.org/ has upgraded so Amplabs jenkins is a year or so behind. I am happy to help with this project if it is something that people think is worthwhile.
Thanks Xin
