I agree. We should still be diligent in keeping dependencies up-to-date as long as the changes are fairly non-destructive (I hear you about guava). That would be a pretty big service to not only users but also developers of libraries on hadoop.
On Fri, Mar 28, 2014 at 11:04 AM, Steve Loughran <ste...@hortonworks.com>wrote: > On 28 March 2014 17:00, Sandy Ryza <sandy.r...@cloudera.com> wrote: > > > My understanding is that unfortunately we're stuck with these for the > rest > > of 2.x, because changing them could break jobs that rely on them. For > jobs > > that want to use newer versions, the recommendation is to use > > mapreduce.user.classpath.first or turn on classpath isolation with > > mapreduce.job.classloader. > > > > > If you look at the compatibility statement of hadoop it makes clear there > are no guarantees about dependencies and especially transitive ones. > > > http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common/Compatibility.html#Java_Classpath > > That is not an accident -any statement of stability would risk painting us > into a corner and never able to update anything. It also allowed us to ship > hadoop 2.2 without doing a rush update of every dependency -that would only > have caused chaos and delays. Protubf -> 2.5 was enough -and that was done > because it was even worse than guava in terms of compatibility policy. > > The issue goes beyond MR as YARN apps pick up the core binaries, so are > constrained by what comes in hadoop/lib. hdfs/lib and yarn/lib. most of > which is pretty dated. > > and if you have an app like hbase or accumulo, with more current > dependencies, you have to play games excluding all of hadoop's > dependencies. But even that doesn't help with Guava, because it is so > aggressive about retiring classes and methods. > > for the sake of our own code and more modern apps, I'm in favour of > gradually rolling out updates that don't break things, force a move to > java7 or require changes to the source. > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
