I guess upgrading the minimal version should also mean cleaning up the codebase, i.e. removing code segments that have been around to allow support for older versions. The overall goal should be to improve the Flink codebase in my opinion. Considering what David said in the old thread about Hadoop users usually lacking behind with version upgrades [1], would we do this version bump in two phases, i.e. adding some deprecation notes and doing the actual cleanup later on? I think Gabor has a point with it not being really mentioned anywhere in the docs (the only location I could find in the docs about Hadoop version is [2]). In this sense, the support for older Hadoop versions was kind of implicit: We're talking about compiling Flink with Hadoop 2.8.5 but also mention older Hadoop versions which leaves room for interpretation.
Additionally, having code that hasn't been touch for a while increases the risk of it Matthias [1] https://lists.apache.org/thread/w7www13tossxrxo1mttgb68v81rf6fks [2] https://nightlies.a1pache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions> On Fri, Oct 21, 2022 at 4:13 AM Xintong Song <tonysong...@gmail.com> wrote: > I believe there are some reflection based approaches in the `flink-yarn` > module, for supporting outdated APIs in early Hadoop versions. > > I haven't done a thorough check, and these are what I get. > - AMRMClientAsyncReflector > - ApplicationSubmissionContextReflector > - ContainerRequestReflector > - RegisterApplicationMasterResponseReflector > - ResourceInformationReflector > > Are we removing these as well? If yes, then Flink can no longer work with > the old hadoop versions. (That's how I understand "bumping the minimal > supported hadoop version".) I personally am not super eager to get rid of > theses, because the relevant parts of codes are no longer frequently > changing, thus the maintenance overhead is low. > > Best, > > Xintong > > > > On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <danrtsey...@gmail.com> wrote: > > > Given that we do not bundle any hadoop classes in the Flink binary, do > you > > mean simply bump the hadoop version in the parent pom? > > If it is, why do not we use the latest stable hadoop version 3.3.4? It > > seems that our cron build has verified that hadoop3 could work. > > > > Best, > > Yang > > > > David Morávek <david.mora...@gmail.com> 于2022年10月19日周三 16:29写道： > > > > > +1; anything below 2.10.x seems to be EOL > > > > > > Best, > > > D. > > > > > > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi < > > balassi.mar...@gmail.com> > > > wrote: > > > > > > > Hi Martjin, > > > > > > > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to > > > > implement the bump? > > > > > > > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi < > > gabor.g.somo...@gmail.com> > > > > wrote: > > > > > > > > > Hi Martin, > > > > > > > > > > Thanks for bringing this up! Lately I was thinking about to bump > the > > > > hadoop > > > > > version to at least 2.6.1 to clean up issues like this: > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159 > > > > > > > > > > All in all +1 from my perspective. > > > > > > > > > > Just a question here. Are we stating the minimum Hadoop version for > > > users > > > > > somewhere in the doc or they need to find it out from source code > > like > > > > > this? > > > > > > > > > > > > > > > > > > > > https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113 > > > > > > > > > > BR, > > > > > G > > > > > > > > > > > > > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser < > > > martijnvis...@apache.org> > > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Little over a year ago a discussion thread was opened on changing > > the > > > > > > minimal supported version of Hadoop and bringing that to 2.8.5. > [1] > > > In > > > > > this > > > > > > discussion thread, I would like to propose to bring that minimal > > > > > supported > > > > > > version of Hadoop to 2.10.2. > > > > > > > > > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified > > as > > > > > > Critical. [2] [3]. While Flink is not directly impacted by those, > > we > > > do > > > > > see > > > > > > vulnerability scanners flag Flink as being vulnerable. We could > > > easily > > > > > > mitigate that by bumping the minimal supported version of Hadoop > to > > > > > 2.10.2. > > > > > > > > > > > > I'm looking forward to your opinions on this topic. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > https://twitter.com/MartijnVisser82 > > > > > > https://github.com/MartijnVisser > > > > > > > > > > > > [1] > > https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5 > > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168 > > > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612 > > > > > > > > > > > > > > > > > > > > >
