Following up on ecosystem, I just took a look at the Apache trunk pom.xml files for HBase, Flume and Oozie. All are specifying 1.6 for source and target in the maven-compiler-plugin configuration, so there may be additional follow-up required here. (For example, if HBase has made a statement that its client will continue to support JDK6, then it wouldn't be practical for them to link to a JDK7 version of hadoop-common.)
+1 for the whole plan though. We can work through these details. Chris Nauroth Hortonworks http://hortonworks.com/ On Fri, Jun 27, 2014 at 3:10 PM, Karthik Kambatla <ka...@cloudera.com> wrote: > +1 to making 2.6 the last JDK6 release. > > If we want, 2.7 could be a parallel release or one soon after 2.6. We could > upgrade other dependencies that require JDK7 as well. > > > On Fri, Jun 27, 2014 at 3:01 PM, Arun C. Murthy <a...@hortonworks.com> > wrote: > > > Thanks everyone for the discussion. Looks like we have come to a > pragmatic > > and progressive conclusion. > > > > In terms of execution of the consensus plan, I think a little bit of > > caution is in order. > > > > Let's give downstream projects more of a runway. > > > > I propose we inform HBase, Pig, Hive etc. that we are considering making > > 2.6 (not 2.5) the last JDK6 release and solicit their feedback. Once they > > are comfortable we can pull the trigger in 2.7. > > > > thanks, > > Arun > > > > > > > On Jun 27, 2014, at 11:34 AM, Karthik Kambatla <ka...@cloudera.com> > > wrote: > > > > > > As someone else already mentioned, we should announce one future > release > > > (may be, 2.5) as the last JDK6-based release before making the move to > > JDK7. > > > > > > I am comfortable calling 2.5 the last JDK6 release. > > > > > > > > > On Fri, Jun 27, 2014 at 11:26 AM, Andrew Wang < > andrew.w...@cloudera.com> > > > wrote: > > > > > >> Hi all, responding to multiple messages here, > > >> > > >> Arun, thanks for the clarification regarding MR classpaths. It sounds > > like > > >> the story there is improved and still improving. > > >> > > >> However, I think we still suffer from this at least on the HDFS side. > We > > >> have a single JAR for all of HDFS, and our clients need to have all > the > > fun > > >> deps like Guava on the classpath. I'm told Spark sticks a newer Guava > at > > >> the front of the classpath and the HDFS client still works okay, but > > this > > >> is more happy coincidence than anything else. While we're leaking > deps, > > >> we're in a scary situation. > > >> > > >> API compat to me means that an app should be able to run on a new > minor > > >> version of Hadoop and not have anything break. MAPREDUCE-4421 sounds > > like > > >> it allows you to run e.g. 2.3 MR jobs on a 2.4 YARN cluster, but what > > >> should also be possible is running an HDFS 2.3 app with HDFS 2.4 JARs > > and > > >> have nothing break. If we muck with the classpath, my understanding is > > that > > >> this could break. > > >> > > >> Owen, bumping the minimum JDK version in a minor release like this > > should > > >> be a one-time exception as Tucu stated. A number of people have > pointed > > out > > >> how painful a forced JDK upgrade is for end users, and it's not > > something > > >> we should be springing on them in a minor release unless we're *very* > > >> confident like in this case. > > >> > > >> Chris, thanks for bringing up the ecosystem. For CDH5, we standardized > > on > > >> JDK7 across the CDH stack, so I think that's an indication that most > > >> ecosystem projects are ready to make the jump. Is that sufficient in > > your > > >> mind? > > >> > > >> For the record, I'm also +1 on the Tucu plan. Is it too late to do > this > > for > > >> 2.5? I'll offer to help out with some of the mechanics. > > >> > > >> Thanks, > > >> Andrew > > >> > > >> On Wed, Jun 25, 2014 at 4:18 PM, Chris Nauroth < > > cnaur...@hortonworks.com> > > >> wrote: > > >> > > >>> I understood the plan for avoiding JDK7-specific features in our > code, > > >> and > > >>> your suggestion to add an extra Jenkins job is a great way to guard > > >> against > > >>> that. The thing I haven't seen discussed yet is how downstream > > projects > > >>> will continue to consume our built artifacts. If a downstream > project > > >>> upgrades to pick up a bug fix, and the jar switches to 1.7 class > files, > > >> but > > >>> their project is still building with 1.6, then it would be a nasty > > >>> surprise. > > >>> > > >>> These are the options I see: > > >>> > > >>> 1. Make sure all other projects upgrade first. This doesn't sound > > >>> feasible, unless all other ecosystem projects have moved to JDK7 > > already. > > >>> If not, then waiting on a single long pole project would hold up our > > >>> migration indefinitely. > > >>> > > >>> 2. We switch to JDK7, but run javac with -target 1.6 until the whole > > >>> ecosystem upgrades. I find this undesirable, because in a certain > > sense, > > >>> it still leaves a bit of 1.6 lingering in the project. (I'll assume > > that > > >>> end-of-life for JDK6 also means end-of-life for the 1.6 bytecode > > format.) > > >>> > > >>> 3. Just declare a clean break on some version (your earlier email > said > > >> 2.5) > > >>> and start publishing artifacts built with JDK7 and no -target option. > > >>> Overall, this is my preferred option. However, as a side effect, > this > > >>> sets us up for longer-term maintenance and patch releases off of the > > 2.4 > > >>> branch if a downstream project that's still on 1.6 needs to pick up a > > >>> critical bug fix. > > >>> > > >>> Of course, this is all a moot point if all the downstream ecosystem > > >>> projects have already made the switch to JDK7. I don't know the > status > > >> of > > >>> that off the top of my head. Maybe someone else out there knows? If > > >> not, > > >>> then I expect I can free up enough in a few weeks to volunteer for > > >> tracking > > >>> down that information. > > >>> > > >>> Chris Nauroth > > >>> Hortonworks > > >>> http://hortonworks.com/ > > >>> > > >>> > > >>> > > >>> On Wed, Jun 25, 2014 at 3:12 PM, Alejandro Abdelnur < > t...@cloudera.com > > > > > >>> wrote: > > >>> > > >>>> Chris, > > >>>> > > >>>> Compiling with jdk7 and doing javac -target 1.6 is not sufficient, > you > > >>> are > > >>>> still using jdk7 libraries and you could use new APIs, thus breaking > > >> jdk6 > > >>>> both at compile and runtime. > > >>>> > > >>>> you need to compile with jdk6 to ensure you are not running into > that > > >>>> scenario. that is why i was suggesting the nightly jdk6 build/test > > >>> jenkins > > >>>> job. > > >>>> > > >>>> > > >>>> On Wed, Jun 25, 2014 at 2:04 PM, Chris Nauroth < > > >> cnaur...@hortonworks.com > > >>>> > > >>>> wrote: > > >>>> > > >>>>> I'm also +1 for getting us to JDK7 within the 2.x line after > reading > > >>> the > > >>>>> proposals and catching up on the discussion in this thread. > > >>>>> > > >>>>> Has anyone yet considered how to coordinate this change with > > >> downstream > > >>>>> projects? Would we request downstream projects to upgrade to JDK7 > > >>> first > > >>>>> before we make the move? Would we switch to JDK7, but run javac > > >>> -target > > >>>>> 1.6 to maintain compatibility for downstream projects during an > > >> interim > > >>>>> period? > > >>>>> > > >>>>> Chris Nauroth > > >>>>> Hortonworks > > >>>>> http://hortonworks.com/ > > >>>>> > > >>>>> > > >>>>> > > >>>>>> On Wed, Jun 25, 2014 at 9:48 AM, Owen O'Malley < > omal...@apache.org> > > >>>>> wrote: > > >>>>> > > >>>>>> On Tue, Jun 24, 2014 at 4:44 PM, Alejandro Abdelnur < > > >>> t...@cloudera.com > > >>>>> > > >>>>>> wrote: > > >>>>>> > > >>>>>>> After reading this thread and thinking a bit about it, I think it > > >>>>> should > > >>>>>> be > > >>>>>>> OK such move up to JDK7 in Hadoop > > >>>>>> > > >>>>>> > > >>>>>> I agree with Alejandro. Changing minimum JDKs is not an > > >> incompatible > > >>>>> change > > >>>>>> and is fine in the 2 branch. (Although I think it is would *not* > be > > >>>>>> appropriate for a patch release.) Of course we need to do it with > > >>>>>> forethought and testing, but moving off of JDK 6, which is EOL'ed > > >> is > > >>> a > > >>>>> good > > >>>>>> thing. Moving to Java 8 as a minimum seems much too aggressive and > > >> I > > >>>>> would > > >>>>>> push back on that. > > >>>>>> > > >>>>>> I'm also think that we need to let the dust settle on the Hadoop 2 > > >>> line > > >>>>> for > > >>>>>> a while before we talk about Hadoop 3. It seems that it has only > > >> been > > >>>> in > > >>>>>> the last 6 months that Hadoop 2 adoption has reached the main > > >> stream > > >>>>> users. > > >>>>>> Our user community needs time to digest the changes in Hadoop 2.x > > >>>> before > > >>>>> we > > >>>>>> fracture the community by starting to discuss Hadoop 3 releases. > > >>>>>> > > >>>>>> .. Owen > > >>>>> > > >>>>> -- > > >>>>> CONFIDENTIALITY NOTICE > > >>>>> NOTICE: This message is intended for the use of the individual or > > >>> entity > > >>>> to > > >>>>> which it is addressed and may contain information that is > > >> confidential, > > >>>>> privileged and exempt from disclosure under applicable law. If the > > >>> reader > > >>>>> of this message is not the intended recipient, you are hereby > > >> notified > > >>>> that > > >>>>> any printing, copying, dissemination, distribution, disclosure or > > >>>>> forwarding of this communication is strictly prohibited. If you > have > > >>>>> received this communication in error, please contact the sender > > >>>> immediately > > >>>>> and delete it from your system. Thank You. > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Alejandro > > >>> > > >>> -- > > >>> CONFIDENTIALITY NOTICE > > >>> NOTICE: This message is intended for the use of the individual or > > entity > > >> to > > >>> which it is addressed and may contain information that is > confidential, > > >>> privileged and exempt from disclosure under applicable law. If the > > reader > > >>> of this message is not the intended recipient, you are hereby > notified > > >> that > > >>> any printing, copying, dissemination, distribution, disclosure or > > >>> forwarding of this communication is strictly prohibited. If you have > > >>> received this communication in error, please contact the sender > > >> immediately > > >>> and delete it from your system. Thank You. > > >> > > > > -- > > CONFIDENTIALITY NOTICE > > NOTICE: This message is intended for the use of the individual or entity > to > > which it is addressed and may contain information that is confidential, > > privileged and exempt from disclosure under applicable law. If the reader > > of this message is not the intended recipient, you are hereby notified > that > > any printing, copying, dissemination, distribution, disclosure or > > forwarding of this communication is strictly prohibited. If you have > > received this communication in error, please contact the sender > immediately > > and delete it from your system. Thank You. > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
