I think we need some way of isolating YARN, MR, and HDFS clients from the Hadoop dependencies. Anything else just isn't sane... whatever we may say, there will always be clients that rely on the dependencies that we pull in, if we make those visible. I can't really blame clients for this. It's sort of unclear what happens when you pull in multiple versions of a dependency through Maven.
The easiest way to solve this would be to use something like Maven-shade. best, Colin On Fri, Mar 28, 2014 at 10:30 AM, Steve Loughran <ste...@hortonworks.com> wrote: > I don't disagree about version age -I've just been downgrading a project to > use some of the versions. The issue with YARN apps is that you get these on > your classpath. > > > 1. There's a JIRA: https://issues.apache.org/jira/browse/HADOOP-9991 > 2. Nobody is working full time on these, I sporadically apply the > patches -I get to feel the pain downstream > > commons lang is at 2.6 now, which should keep you happy. > > jetty? Now that the shuffle is on netty we could think about this and jersey > > Guava is trouble. Leave it: new code doesn't work. Remove it and old code > stops working. > https://issues.apache.org/jira/browse/HADOOP-10101 > > I think for branch-3 we should go ahead an apply the patch. For branch 2, > it's too destructuve. > > At some point we also have to commit being java7 + only -this would > actually help us move up to some of the later dependencies. That's clearly > not a branch-2 > > > > > On 28 March 2014 14:59, Sangjin Lee <sjl...@gmail.com> wrote: > >> Hi folks, >> >> Even as 2.3 was released, several dependencies of hadoop are quite dated. >> And more and more of them are causing friction for hadoop-related libraries >> and hadoop users in general, as these dependency versions are often >> incompatibly different than the versions most people use these days. So >> this is becoming a very real problem, if not one already. >> >> Some key ones include >> - guava: 11.0.2 (current 16.0.1) >> - jetty: 6.1.26 (current 9.1.3) >> - commons-lang: 2.6 (current 3.3.1) >> >> In particular, guava is already causing a lot of issues as many developers >> have moved onto a newer version and are using APIs that do not exist in >> 11.0.2, and get NoSuchMethodErrors and such. >> >> Also, for jetty, 6.1.26 has been EOFed for some time now. >> >> Do we have a plan to review some of these dependencies and upgrade them to >> reasonably up-to-date versions? I remember seeing some JIRAs on specific >> dependencies, but has a review been done to determine a good set of >> versions to upgrade to? >> >> It would be great if we could upgrade some of the more common ones at least >> to modern versions. >> >> Thanks, >> Sangjin >> > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You.
