Thank you for your opinion! Sure. I know that history and totally agree with all your concerns. I indeed has hesitated about sending this kind of suggestion for a while.
If Travis CI cannot handle those simple jobs at this time again, we must turn off from Spark PR queue. We can see the result quickly in one or two days. To turn on/off, Spark have nothing to do. INFRA team will do that. In fact, the goal is not about using another CI (like Travis), it is about preventing the followings. 1. JDK7 compilation errors. (Recently, 2 days ago and 5 days ago) 2. Java static errors. (Not critical but more frequently.) 3. Maven installation errors. (A month ago, it's reported in this mailing list.) Scala 2.10 compilation errors are fixed nearly instantly. But, 1~3 were not. If SparkPullRequestBuilder can do the above 1~3, that's the best for us. Do you think it is possible in some ways? By the way, as of today, Spark has 724 Java files and 96762 lines (without comment/blank). It's about 1/3 of Scala code. It's not small. -------------------------------------------------------------------------- Language files blank comment code -------------------------------------------------------------------------- Scala 2368 63578 124904 322518 Java 724 18569 23445 96762 Dongjoon. On Mon, May 23, 2016 at 12:20 PM, Michael Armbrust <mich...@databricks.com> wrote: > We did turn on travis a few years ago, but ended up turning it off because > it was failing (I believe because of insufficient resources) which was > confusing for developers. I wouldn't be opposed to turning it on if it > provides more/faster signal, but its not obvious to me that it would. In > particular, do we know that given the rate PRs are created if we will hit > rate limits? > > Really my main feedback is, if the java linter is important we should > probably have it as part of the canonical build process. I worry about > having more than one set of CI infrastructure to maintain. > > On Mon, May 23, 2016 at 9:43 AM, Dongjoon Hyun <dongj...@apache.org> > wrote: > >> Thank you, Steve and Hyukjin. >> >> And, don't worry, Ted. >> >> Travis launches new VMs for every PR. >> >> Apache Spark repository uses the following setting. >> >> VM: Google Compute Engine >> OS: Ubuntu 14.04.3 LTS Server Edition 64bit >> CPU: ~2 CORE >> RAM: 7.5GB >> >> FYI, you can find more information about this here. >> >> >> https://docs.travis-ci.com/user/ci-environment/#Virtualization-environments >> >> Dongjoon. >> >> >> >> On Mon, May 23, 2016 at 6:32 AM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Do you know if more than one PR would be verified on the same machine ? >>> >>> I wonder whether the 'mvn install' from two simultaneous PR builds may >>> have conflict. >>> >>> On Sun, May 22, 2016 at 9:21 PM, Dongjoon Hyun <dongj...@apache.org> >>> wrote: >>> >>>> Thank you for feedback. Sure, correctly, that's the reason why the >>>> current SparkPullRequestBuilder do not run `lint-java`. :-) >>>> >>>> In addition, that's the same reason why contributors are reluctant to >>>> run `lint-java` and causes breaking on JDK7 builds. >>>> >>>> Such a tedious and time-consuming job should be done by CI without >>>> human interventions. >>>> >>>> By the way, why do you think we need to wait for that? We should not >>>> wait for any CIs, we should continue our own work. >>>> >>>> My proposal isn't for making you wait to watch the result. There are >>>> two use cases I want for us to focus here. >>>> >>>> Case 1: When you make a PR to Spark PR queue. >>>> >>>> Travis CI will finish before SparkPullRequestBuilder. >>>> We will run the followings in parallel mode. >>>> 1. Current SparkPullRequestBuilder: JDK8 + sbt build + (no >>>> Java Linter) >>>> 2. Travis: JDK7 + mvn build + Java Linter >>>> 3. Travis: JDK8 + mvn build + Java Linter >>>> As we know, 1 is the longest time-consuming one which have lots of >>>> works (except maven building or lint- java). You don't need to wait more >>>> in many cases. Yes, in many cases, not all the cases. >>>> >>>> >>>> Case 2: When you prepare a PR on your branch. >>>> >>>> If you are at the final commit (maybe already-squashed), just go to >>>> case 1. >>>> >>>> However, usually, we makes lots of commits locally while making >>>> preparing our PR. >>>> And, finally we squashed them into one and send a PR to Spark. >>>> I mean you can use Travis CI during preparing your PRs. >>>> Again, don't wait for Travis CI. Just push it sometime or at every >>>> commit, and continue your work. >>>> >>>> At the final stage when you finish your coding, squash your commits >>>> into one, >>>> and amend your commit title or messages, see the Travis CI. >>>> Or, you can monitor Travis CI result on status menu bar. >>>> If it shows green icon, you have nothing to do. >>>> >>>> https://docs.travis-ci.com/user/apps/ >>>> >>>> To sum up, I think we don't need to wait for any CIs. It's like an >>>> email. `Send and back to work.` >>>> >>>> Dongjoon. >>>> >>>> >>>> On Sun, May 22, 2016 at 8:32 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> Without Zinc, 'mvn -DskipTests clean install' takes ~30 minutes. >>>>> >>>>> Maybe not everyone is willing to wait that long. >>>>> >>>>> On Sun, May 22, 2016 at 1:30 PM, Dongjoon Hyun <dongj...@apache.org> >>>>> wrote: >>>>> >>>>>> Oh, Sure. My bad! >>>>>> >>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>> - For Oracle JDK8, mvn -DskipTests install and run `dev/lint-java`. >>>>>> >>>>>> Thank you, Ted. >>>>>> >>>>>> Dongjoon. >>>>>> >>>>>> On Sun, May 22, 2016 at 1:29 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>>> The following line was repeated twice: >>>>>>> >>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>> >>>>>>> Did you intend to cover JDK 8 ? >>>>>>> >>>>>>> Cheers >>>>>>> >>>>>>> On Sun, May 22, 2016 at 1:25 PM, Dongjoon Hyun <dongj...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, All. >>>>>>>> >>>>>>>> I want to propose the followings. >>>>>>>> >>>>>>>> - Turn on Travis CI for Apache Spark PR queue. >>>>>>>> - Recommend this for contributors, too >>>>>>>> >>>>>>>> Currently, Spark provides Travis CI configuration file to help >>>>>>>> contributors check Scala/Java style conformance and JDK7/8 compilation >>>>>>>> easily during their preparing pull requests. Please note that it's only >>>>>>>> about static analysis. >>>>>>>> >>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>> - For Oracle JDK7, mvn -DskipTests install and run `dev/lint-java`. >>>>>>>> Scalastyle is included in the step 'mvn install', too. >>>>>>>> >>>>>>>> Yep, if you turn on your Travis CI configuration, you can already >>>>>>>> see the results on your branches before making PR. I wrote this email >>>>>>>> to >>>>>>>> prevent more failures proactively and community-widely. >>>>>>>> >>>>>>>> For stability, I have been monitoring that for two weeks. It >>>>>>>> detects the failures or recovery on JDK7 builds or Java linter on Spark >>>>>>>> master branch correctly. The only exceptional case I observed rarely is >>>>>>>> `timeout` failure due to hangs of maven. But, as we know, it's happen >>>>>>>> in >>>>>>>> our Jenkins SparkPullRequestBuilder, too. I think we can ignore that. >>>>>>>> >>>>>>>> I'm sure that this will save much more community's efforts on the >>>>>>>> static errors by preventing them at the very early stage. But, there >>>>>>>> might >>>>>>>> be another reason not to do this. I'm wondering about your thoughts. >>>>>>>> >>>>>>>> I can make a Apache INFRA Jira issue for this if there is some >>>>>>>> consensus. >>>>>>>> >>>>>>>> Warmly, >>>>>>>> Dongjoon. >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
