I think that a backup plan could be to announce that JDK7 is deprecated in Spark 2.0 and support for it will be fully removed in Spark 2.1. This gives admins enough warning to install JDK8 along side their "main" JDK (or fully migrate to it), while allowing the project to merge JDK8-specific changes to trunk right after the 2.0 release.
However, I personally think it is better to drop JDK7 now. I'm sure that both the community and the distributors (Databricks, Cloudera, Hortonworks, MapR, IBM etc) will all rush to help their customers migrate their environment to support Spark 2.0, so I think any backlash won't be dramatic or lasting. Just my two cents, Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io On Mon, Apr 4, 2016 at 6:48 PM, Luciano Resende <luckbr1...@gmail.com> wrote: > Reynold, > > Considering the performance improvements you mentioned in your original > e-mail and also considering that few other big data projects have already > or are in progress of abandoning JDK 7, I think it would benefit Spark if > we go with JDK 8.0 only. > > Are there users that will be less aggressive ? Yes, but those would most > likely be in more stable releases like 1.6.x. > > On Sun, Apr 3, 2016 at 10:28 PM, Reynold Xin <r...@databricks.com> wrote: > >> Since my original email, I've talked to a lot more users and looked at >> what various environments support. It is true that a lot of enterprises, >> and even some technology companies, are still using Java 7. One thing is >> that up until this date, users still can't install openjdk 8 on Ubuntu by >> default. I see that as an indication that it is too early to drop Java 7. >> >> Looking at the timeline, JDK release a major new version roughly every 3 >> years. We dropped Java 6 support one year ago, so from a timeline point of >> view we would be very aggressive here if we were to drop Java 7 support in >> Spark 2.0. >> >> Note that not dropping Java 7 support now doesn't mean we have to support >> Java 7 throughout Spark 2.x. We dropped Java 6 support in Spark 1.5, even >> though Spark 1.0 started with Java 6. >> >> In terms of testing, Josh has actually improved our test infra so now we >> would run the Java 8 tests: https://github.com/apache/spark/pull/12073 >> >> >> >> >> On Thu, Mar 24, 2016 at 8:51 PM, Liwei Lin <lwl...@gmail.com> wrote: >> >>> Arguments are really convincing; new Dataset API as well as performance >>> >>> improvements is exiting, so I'm personally +1 on moving onto Java8. >>> >>> >>> >>> However, I'm afraid Tencent is one of "the organizations stuck with >>> Java7" >>> >>> -- our IT Infra division wouldn't upgrade to Java7 until Java8 is out, >>> and >>> >>> wouldn't upgrade to Java8 until Java9 is out. >>> >>> >>> So: >>> >>> (non-binding) +1 on dropping scala 2.10 support >>> >>> (non-binding) -1 on dropping Java 7 support >>> >>> * as long as we figure out a practical way to run >>> Spark with >>> >>> JDK8 on JDK7 clusters, this -1 would then >>> definitely be +1 >>> >>> >>> Thanks ! >>> >>> On Fri, Mar 25, 2016 at 10:28 AM, Koert Kuipers <ko...@tresata.com> >>> wrote: >>> >>>> i think that logic is reasonable, but then the same should also apply >>>> to scala 2.10, which is also unmaintained/unsupported at this point >>>> (basically has been since march 2015 except for one hotfix due to a license >>>> incompatibility) >>>> >>>> who wants to support scala 2.10 three years after they did the last >>>> maintenance release? >>>> >>>> >>>> On Thu, Mar 24, 2016 at 9:59 PM, Mridul Muralidharan <mri...@gmail.com> >>>> wrote: >>>> >>>>> Removing compatibility (with jdk, etc) can be done with a major >>>>> release- given that 7 has been EOLed a while back and is now unsupported, >>>>> we have to decide if we drop support for it in 2.0 or 3.0 (2+ years from >>>>> now). >>>>> >>>>> Given the functionality & performance benefits of going to jdk8, >>>>> future enhancements relevant in 2.x timeframe ( scala, dependencies) which >>>>> requires it, and simplicity wrt code, test & support it looks like a good >>>>> checkpoint to drop jdk7 support. >>>>> >>>>> As already mentioned in the thread, existing yarn clusters are >>>>> unaffected if they want to continue running jdk7 and yet use >>>>> spark2 (install jdk8 on all nodes and use it via JAVA_HOME, or worst case >>>>> distribute jdk8 as archive - suboptimal). >>>>> I am unsure about mesos (standalone might be easier upgrade I guess ?). >>>>> >>>>> >>>>> Proposal is for 1.6x line to continue to be supported with critical >>>>> fixes; newer features will require 2.x and so jdk8 >>>>> >>>>> Regards >>>>> Mridul >>>>> >>>>> >>>>> On Thursday, March 24, 2016, Marcelo Vanzin <van...@cloudera.com> >>>>> wrote: >>>>> >>>>>> On Thu, Mar 24, 2016 at 4:50 PM, Reynold Xin <r...@databricks.com> >>>>>> wrote: >>>>>> > If you want to go down that route, you should also ask somebody who >>>>>> has had >>>>>> > experience managing a large organization's applications and try to >>>>>> update >>>>>> > Scala version. >>>>>> >>>>>> I understand both sides. But if you look at what I've been asking >>>>>> since the beginning, it's all about the cost and benefits of dropping >>>>>> support for java 1.7. >>>>>> >>>>>> The biggest argument in your original e-mail is about testing. And the >>>>>> testing cost is much bigger for supporting scala 2.10 than it is for >>>>>> supporting java 1.7. If you read one of my earlier replies, it should >>>>>> be even possible to just do everything in a single job - compile for >>>>>> java 7 and still be able to test things in 1.8, including lambdas, >>>>>> which seems to be the main thing you were worried about. >>>>>> >>>>>> >>>>>> > On Thu, Mar 24, 2016 at 4:48 PM, Marcelo Vanzin < >>>>>> van...@cloudera.com> wrote: >>>>>> >> >>>>>> >> On Thu, Mar 24, 2016 at 4:46 PM, Reynold Xin <r...@databricks.com> >>>>>> wrote: >>>>>> >> > Actually it's *way* harder to upgrade Scala from 2.10 to 2.11, >>>>>> than >>>>>> >> > upgrading the JVM runtime from 7 to 8, because Scala 2.10 and >>>>>> 2.11 are >>>>>> >> > not >>>>>> >> > binary compatible, whereas JVM 7 and 8 are binary compatible >>>>>> except >>>>>> >> > certain >>>>>> >> > esoteric cases. >>>>>> >> >>>>>> >> True, but ask anyone who manages a large cluster how long it would >>>>>> >> take them to upgrade the jdk across their cluster and validate all >>>>>> >> their applications and everything... binary compatibility is a tiny >>>>>> >> drop in that bucket. >>>>>> >> >>>>>> >> -- >>>>>> >> Marcelo >>>>>> > >>>>>> > >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Marcelo >>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >>>>>> For additional commands, e-mail: dev-h...@spark.apache.org >>>>>> >>>>>> >>>> >>> >> > > > -- > Luciano Resende > http://twitter.com/lresende1975 > http://lresende.blogspot.com/ >
