> Additionally, having code that hasn't been touch for a while increases the risk of it Sorry about this incomplete confusing sentence. I was about to remove it when accidentally pushing the shortcut for sending the message out to the mailing list.
On Tue, Oct 25, 2022 at 10:18 AM Matthias Pohl <matthias.p...@aiven.io> wrote: > I guess upgrading the minimal version should also mean cleaning up the > codebase, i.e. removing code segments that have been around to allow > support for older versions. The overall goal should be to improve the Flink > codebase in my opinion. Considering what David said in the old thread about > Hadoop users usually lacking behind with version upgrades [1], would we do > this version bump in two phases, i.e. adding some deprecation notes and > doing the actual cleanup later on? I think Gabor has a point with it not > being really mentioned anywhere in the docs (the only location I could find > in the docs about Hadoop version is [2]). In this sense, the support for > older Hadoop versions was kind of implicit: We're talking about compiling > Flink with Hadoop 2.8.5 but also mention older Hadoop versions which leaves > room for interpretation. > > Additionally, having code that hasn't been touch for a while increases the > risk of it > > Matthias > > [1] https://lists.apache.org/thread/w7www13tossxrxo1mttgb68v81rf6fks > [2] > https://nightlies.a1pache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions > <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions> > > On Fri, Oct 21, 2022 at 4:13 AM Xintong Song <tonysong...@gmail.com> > wrote: > >> I believe there are some reflection based approaches in the `flink-yarn` >> module, for supporting outdated APIs in early Hadoop versions. >> >> I haven't done a thorough check, and these are what I get. >> - AMRMClientAsyncReflector >> - ApplicationSubmissionContextReflector >> - ContainerRequestReflector >> - RegisterApplicationMasterResponseReflector >> - ResourceInformationReflector >> >> Are we removing these as well? If yes, then Flink can no longer work with >> the old hadoop versions. (That's how I understand "bumping the minimal >> supported hadoop version".) I personally am not super eager to get rid of >> theses, because the relevant parts of codes are no longer frequently >> changing, thus the maintenance overhead is low. >> >> Best, >> >> Xintong >> >> >> >> On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <danrtsey...@gmail.com> wrote: >> >> > Given that we do not bundle any hadoop classes in the Flink binary, do >> you >> > mean simply bump the hadoop version in the parent pom? >> > If it is, why do not we use the latest stable hadoop version 3.3.4? It >> > seems that our cron build has verified that hadoop3 could work. >> > >> > Best, >> > Yang >> > >> > David Morávek <david.mora...@gmail.com> 于2022年10月19日周三 16:29写道： >> > >> > > +1; anything below 2.10.x seems to be EOL >> > > >> > > Best, >> > > D. >> > > >> > > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi < >> > balassi.mar...@gmail.com> >> > > wrote: >> > > >> > > > Hi Martjin, >> > > > >> > > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to >> > > > implement the bump? >> > > > >> > > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi < >> > gabor.g.somo...@gmail.com> >> > > > wrote: >> > > > >> > > > > Hi Martin, >> > > > > >> > > > > Thanks for bringing this up! Lately I was thinking about to bump >> the >> > > > hadoop >> > > > > version to at least 2.6.1 to clean up issues like this: >> > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159 >> > > > > >> > > > > All in all +1 from my perspective. >> > > > > >> > > > > Just a question here. Are we stating the minimum Hadoop version >> for >> > > users >> > > > > somewhere in the doc or they need to find it out from source code >> > like >> > > > > this? >> > > > > >> > > > > >> > > > >> > > >> > >> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113 >> > > > > >> > > > > BR, >> > > > > G >> > > > > >> > > > > >> > > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser < >> > > martijnvis...@apache.org> >> > > > > wrote: >> > > > > >> > > > > > Hi everyone, >> > > > > > >> > > > > > Little over a year ago a discussion thread was opened on >> changing >> > the >> > > > > > minimal supported version of Hadoop and bringing that to 2.8.5. >> [1] >> > > In >> > > > > this >> > > > > > discussion thread, I would like to propose to bring that minimal >> > > > > supported >> > > > > > version of Hadoop to 2.10.2. >> > > > > > >> > > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are >> classified >> > as >> > > > > > Critical. [2] [3]. While Flink is not directly impacted by >> those, >> > we >> > > do >> > > > > see >> > > > > > vulnerability scanners flag Flink as being vulnerable. We could >> > > easily >> > > > > > mitigate that by bumping the minimal supported version of >> Hadoop to >> > > > > 2.10.2. >> > > > > > >> > > > > > I'm looking forward to your opinions on this topic. >> > > > > > >> > > > > > Best regards, >> > > > > > >> > > > > > Martijn >> > > > > > https://twitter.com/MartijnVisser82 >> > > > > > https://github.com/MartijnVisser >> > > > > > >> > > > > > [1] >> > https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5 >> > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168 >> > > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612 >> > > > > > >> > > > > >> > > > >> > > >> > >> >
