I believe there are some reflection based approaches in the `flink-yarn` module, for supporting outdated APIs in early Hadoop versions.
I haven't done a thorough check, and these are what I get. - AMRMClientAsyncReflector - ApplicationSubmissionContextReflector - ContainerRequestReflector - RegisterApplicationMasterResponseReflector - ResourceInformationReflector Are we removing these as well? If yes, then Flink can no longer work with the old hadoop versions. (That's how I understand "bumping the minimal supported hadoop version".) I personally am not super eager to get rid of theses, because the relevant parts of codes are no longer frequently changing, thus the maintenance overhead is low. Best, Xintong On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <danrtsey...@gmail.com> wrote: > Given that we do not bundle any hadoop classes in the Flink binary, do you > mean simply bump the hadoop version in the parent pom? > If it is, why do not we use the latest stable hadoop version 3.3.4? It > seems that our cron build has verified that hadoop3 could work. > > Best, > Yang > > David Morávek <david.mora...@gmail.com> 于2022年10月19日周三 16:29写道： > > > +1; anything below 2.10.x seems to be EOL > > > > Best, > > D. > > > > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi < > balassi.mar...@gmail.com> > > wrote: > > > > > Hi Martjin, > > > > > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to > > > implement the bump? > > > > > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi < > gabor.g.somo...@gmail.com> > > > wrote: > > > > > > > Hi Martin, > > > > > > > > Thanks for bringing this up! Lately I was thinking about to bump the > > > hadoop > > > > version to at least 2.6.1 to clean up issues like this: > > > > > > > > > > > > > > https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159 > > > > > > > > All in all +1 from my perspective. > > > > > > > > Just a question here. Are we stating the minimum Hadoop version for > > users > > > > somewhere in the doc or they need to find it out from source code > like > > > > this? > > > > > > > > > > > > > > https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113 > > > > > > > > BR, > > > > G > > > > > > > > > > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser < > > martijnvis...@apache.org> > > > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > Little over a year ago a discussion thread was opened on changing > the > > > > > minimal supported version of Hadoop and bringing that to 2.8.5. [1] > > In > > > > this > > > > > discussion thread, I would like to propose to bring that minimal > > > > supported > > > > > version of Hadoop to 2.10.2. > > > > > > > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified > as > > > > > Critical. [2] [3]. While Flink is not directly impacted by those, > we > > do > > > > see > > > > > vulnerability scanners flag Flink as being vulnerable. We could > > easily > > > > > mitigate that by bumping the minimal supported version of Hadoop to > > > > 2.10.2. > > > > > > > > > > I'm looking forward to your opinions on this topic. > > > > > > > > > > Best regards, > > > > > > > > > > Martijn > > > > > https://twitter.com/MartijnVisser82 > > > > > https://github.com/MartijnVisser > > > > > > > > > > [1] > https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5 > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168 > > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612 > > > > > > > > > > > > > > >
