I want to clarify something here. For Travis CI, it's free for open source projects and there is only one management point, `.travis.xml`, for Spark community.
It's not some like physical Jenkins cluster farm. It's just a cloud service like Github. PS. I'm also not an employee of Travis(or Github). :-) If Spark uses Travis CI freely, they might dislike me for the heavy traffic. On Mon, May 23, 2016 at 1:26 PM, Dongjoon Hyun <dongj...@apache.org> wrote: > Thank you, Shane! > > I really hope that SparkPullRequestBuilder handle them if possible. > > Dongjoon. > > On Mon, May 23, 2016 at 1:24 PM, Dongjoon Hyun <dongj...@apache.org> > wrote: > >> Thank you for your opinion! >> >> Sure. I know that history and totally agree with all your concerns. >> I indeed has hesitated about sending this kind of suggestion for a while. >> >> If Travis CI cannot handle those simple jobs at this time again, >> we must turn off from Spark PR queue. >> We can see the result quickly in one or two days. >> To turn on/off, Spark have nothing to do. INFRA team will do that. >> >> In fact, the goal is not about using another CI (like Travis), it is >> about preventing the followings. >> >> 1. JDK7 compilation errors. (Recently, 2 days ago and 5 days ago) >> 2. Java static errors. (Not critical but more frequently.) >> 3. Maven installation errors. (A month ago, it's reported in this mailing >> list.) >> >> Scala 2.10 compilation errors are fixed nearly instantly. But, 1~3 were >> not. >> If SparkPullRequestBuilder can do the above 1~3, that's the best for us. >> Do you think it is possible in some ways? >> >> By the way, as of today, Spark has 724 Java files and 96762 lines >> (without comment/blank). >> It's about 1/3 of Scala code. It's not small. >> -------------------------------------------------------------------------- >> Language files blank comment code >> -------------------------------------------------------------------------- >> Scala 2368 63578 124904 322518 >> Java 724 18569 23445 >> 96762 >> >> Dongjoon. >> >> >> >> On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust < >> mich...@databricks.com> wrote: >> >>> We did turn on travis a few years ago, but ended up turning it off >>> because it was failing (I believe because of insufficient resources) which >>> was confusing for developers. I wouldn't be opposed to turning it on if it >>> provides more/faster signal, but its not obvious to me that it would. In >>> particular, do we know that given the rate PRs are created if we will hit >>> rate limits? >>> >>> Really my main feedback is, if the java linter is important we should >>> probably have it as part of the canonical build process. I worry about >>> having more than one set of CI infrastructure to maintain. >>> >>> On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <dongj...@apache.org> >>> wrote: >>> >>>> Thank you, Steve and Hyukjin. >>>> >>>> And, don't worry, Ted. >>>> >>>> Travis launches new VMs for every PR. >>>> >>>> Apache Spark repository uses the following setting. >>>> >>>> VM: Google Compute Engine >>>> OS: Ubuntu 14.04.3 LTS Server Edition 64bit >>>> CPU: ~2 CORE >>>> RAM: 7.5GB >>>> >>>> FYI, you can find more information about this here. >>>> >>>> >>>> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments >>>> >>>> Dongjoon. >>>> >>>> >>>> >>>> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> Do you know if more than one PR would be verified on the same machine ? >>>>> >>>>> I wonder whether the 'mvn install' from two simultaneous PR builds may >>>>> have conflict. >>>>> >>>>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <dongj...@apache.org> >>>>> wrote: >>>>> >>>>>> Thank you for feedback. Sure, correctly, that's the reason why the >>>>>> current SparkPullRequestBuilder do not run `lint-java`. :-) >>>>>> >>>>>> In addition, that's the same reason why contributors are reluctant to >>>>>> run `lint-java` and causes breaking on JDK7 builds. >>>>>> >>>>>> Such a tedious and time-consuming job should be done by CI without >>>>>> human interventions. >>>>>> >>>>>> By the way, why do you think we need to wait for that? We should not >>>>>> wait for any CIs, we should continue our own work. >>>>>> >>>>>> My proposal isn't for making you wait to watch the result. There are >>>>>> two use cases I want for us to focus here. >>>>>> >>>>>> Case 1: When you make a PR to Spark PR queue. >>>>>> >>>>>> Travis CI will finish before SparkPullRequestBuilder. >>>>>> We will run the followings in parallel mode. >>>>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no >>>>>> Java Linter) >>>>>> 2. Travis: JDK7 + mvn build + Java Linter >>>>>> 3. Travis: JDK8 + mvn build + Java Linter >>>>>> As we know, 1 is the longest time-consuming one which have lots >>>>>> of works (except maven building or lint- java). You don't need to wait >>>>>> more in many cases. Yes, in many cases, not all the cases. >>>>>> >>>>>> >>>>>> Case 2: When you prepare a PR on your branch. >>>>>> >>>>>> If you are at the final commit (maybe already-squashed), just go >>>>>> to case 1. >>>>>> >>>>>> However, usually, we makes lots of commits locally while making >>>>>> preparing our PR. >>>>>> And, finally we squashed them into one and send a PR to Spark. >>>>>> I mean you can use Travis CI during preparing your PRs. >>>>>> Again, don't wait for Travis CI. Just push it sometime or at >>>>>> every commit, and continue your work. >>>>>> >>>>>> At the final stage when you finish your coding, squash your >>>>>> commits into one, >>>>>> and amend your commit title or messages, see the Travis CI. >>>>>> Or, you can monitor Travis CI result on status menu bar. >>>>>> If it shows green icon, you have nothing to do. >>>>>> >>>>>> https://docs.travis-ci.com/user/apps/ >>>>>> >>>>>> To sum up, I think we don't need to wait for any CIs. It's like an >>>>>> email. `Send and back to work.` >>>>>> >>>>>> Dongjoon. >>>>>> >>>>>> >>>>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes. >>>>>>> >>>>>>> Maybe not everyone is willing to wait that long. >>>>>>> >>>>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <dongj...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Oh, Sure. My bad! >>>>>>>> >>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>> >>>>>>>> Thank you, Ted. >>>>>>>> >>>>>>>> Dongjoon. >>>>>>>> >>>>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yuzhih...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> The following line was repeated twice: >>>>>>>>> >>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>>> >>>>>>>>> Did you intend to cover JDK 8 ? >>>>>>>>> >>>>>>>>> Cheers >>>>>>>>> >>>>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun < >>>>>>>>> dongj...@apache.org> wrote: >>>>>>>>> >>>>>>>>>> Hi, All. >>>>>>>>>> >>>>>>>>>> I want to propose the followings. >>>>>>>>>> >>>>>>>>>> - Turn on Travis CI for Apache Spark PR queue. >>>>>>>>>> - Recommend this for contributors, too >>>>>>>>>> >>>>>>>>>> Currently, Spark provides Travis CI configuration file to help >>>>>>>>>> contributors check Scala/Java style conformance and JDK7/8 >>>>>>>>>> compilation >>>>>>>>>> easily during their preparing pull requests. Please note that it's >>>>>>>>>> only >>>>>>>>>> about static analysis. >>>>>>>>>> >>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run >>>>>>>>>> `dev/lint-java`. >>>>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run >>>>>>>>>> `dev/lint-java`. >>>>>>>>>> Scalastyle is included in the step 'mvn install', too. >>>>>>>>>> >>>>>>>>>> Yep, if you turn on your Travis CI configuration, you can already >>>>>>>>>> see the results on your branches before making PR. I wrote this >>>>>>>>>> email to >>>>>>>>>> prevent more failures proactively and community-widely. >>>>>>>>>> >>>>>>>>>> For stability, I have been monitoring that for two weeks. It >>>>>>>>>> detects the failures or recovery on JDK7 builds or Java linter on >>>>>>>>>> Spark >>>>>>>>>> master branch correctly. The only exceptional case I observed rarely >>>>>>>>>> is >>>>>>>>>> `timeout` failure due to hangs of maven. But, as we know, it's >>>>>>>>>> happen in >>>>>>>>>> our Jenkins SparkPullRequestBuilder, too. I think we can ignore that. >>>>>>>>>> >>>>>>>>>> I'm sure that this will save much more community's efforts on the >>>>>>>>>> static errors by preventing them at the very early stage. But, there >>>>>>>>>> might >>>>>>>>>> be another reason not to do this. I'm wondering about your thoughts. >>>>>>>>>> >>>>>>>>>> I can make a Apache INFRA Jira issue for this if there is some >>>>>>>>>> consensus. >>>>>>>>>> >>>>>>>>>> Warmly, >>>>>>>>>> Dongjoon. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
