FWIW we are updating guava in Spark and Hive at Cloudera. Don't know which Apache version are they going to land, but we'll upstream them for sure.
The guava change is debatable. It's not as critical as others. There are critical vulnerabilities in other dependencies that we have no way but to update to a new major/minor version because we are so far behind. And given the critical nature, I think it is worth the risk and backport to lower maintenance releases is warranted. Moreover, our minor releases are at best 1 per year. That is too slow to respond to a critical vulnerability. On Wed, Mar 11, 2020 at 5:02 PM Igor Dvorzhak <i...@google.com.invalid> wrote: > Generally I'm for updating dependencies, but I think that Hadoop should > stick with semantic versioning and do not make major and > minor dependency updates in subminor releases. > > For example, Hadoop 3.2.1 updated Guava to 27.0-jre, and because of this > Spark 3.0 stuck with Hadoop 3.2.0 - they use Hive 2.3.6 that doesn't > support Guava 27.0-jre. > > It would be better to make dependency upgrades when releasing new > major/minor versions, for example Guava 27.0-jre upgrade was more > appropriate for Hadoop 3.3.0 release than 3.2.1. > > On Tue, Mar 10, 2020 at 3:03 PM Wei-Chiu Chuang > <weic...@cloudera.com.invalid> wrote: > >> I'm not hearing any feedback so far, but I want to suggest: >> >> use hadoop-thirdparty repository to host any dependencies that are known >> to >> break compatibility. >> >> Candidate #1 guava >> Candidate #2 Netty >> Candidate #3 Jetty >> >> in fact, HBase shades these dependencies for the exact same reason. >> >> As an example of the cost of compatibility breakage: we spent the last 6 >> months to backport the guava update change (guava 11 --> 27) throughout >> Cloudera's stack, and after 6 months we are not done yet because we have >> to >> update guava in Hadoop, Hive, Spark ..., and Hadoop, Hive and Spark's >> guava >> is in the classpath of every application. >> >> Thoughts? >> >> On Sat, Mar 7, 2020 at 9:31 AM Wei-Chiu Chuang <weic...@apache.org> >> wrote: >> >> > Hi Hadoop devs, >> > >> > I the past, Hadoop tends to be pretty far behind the latest versions of >> > dependencies. Part of that is due to the fear of the breaking changes >> > brought in by the dependency updates. >> > >> > However, things have changed dramatically over the past few years. With >> > more focus on security vulnerabilities, more vulnerabilities are >> discovered >> > in our dependencies, and users put more pressure on patching Hadoop (and >> > its ecosystem) to use the latest dependency versions. >> > >> > As an example, Jackson-databind had 20 CVEs published in the last year >> > alone. >> > >> https://www.cvedetails.com/product/42991/Fasterxml-Jackson-databind.html?vendor_id=15866 >> > >> > Jetty: 4 CVEs in 2019: >> > >> https://www.cvedetails.com/product/34824/Eclipse-Jetty.html?vendor_id=10410 >> > >> > We can no longer keep Hadoop stay behind. The more we stay behind, the >> > harder it is to update. A good example is Jersey migration 1 -> 2 >> > HADOOP-15984 <https://issues.apache.org/jira/browse/HADOOP-15984> >> contributed >> > by Akira. Jersey 1 is no longer supported. But Jersey 2 migration is >> hard. >> > If any critical vulnerability is found in Jersey 1, it will leave us in >> a >> > bad situation since we can't simply update Jersey version and be done. >> > >> > Hadoop 3 adds new public artifacts that shade these dependencies. We >> > should advocate downstream applications to use the public artifacts to >> > avoid breakage. >> > >> > I'd like to hear your thoughts: are you okay to see Hadoop keep up with >> > the latest dependency updates, or would rather stay behind to ensure >> > compatibility? >> > >> > Coupled with that, I'd like to call for more frequent Hadoop releases >> for >> > the same purpose. IMHO that'll require better infrastructure to assist >> the >> > release work and some rethinking our current Hadoop code structure, like >> > separate each subproject into its own repository and release cadence. >> This >> > can be controversial but I think it'll be good for the project in the >> long >> > run. >> > >> > Thanks, >> > Wei-Chiu >> > >> >
