For those that are interested, I have a draft PR open for this
https://github.com/apache/flink/pull/21128 - Feel free to have a look.

I'm not sure yet why the Flink CI fails, since these tests are passing
locally.

Thanks,

Martijn

On Tue, Oct 25, 2022 at 10:21 AM Matthias Pohl
<matthias.p...@aiven.io.invalid> wrote:

> > Additionally, having code that hasn't been touch for a while increases
> the risk of it
> Sorry about this incomplete confusing sentence. I was about to remove it
> when accidentally pushing the shortcut for sending the message out to the
> mailing list.
>
> On Tue, Oct 25, 2022 at 10:18 AM Matthias Pohl <matthias.p...@aiven.io>
> wrote:
>
> > I guess upgrading the minimal version should also mean cleaning up the
> > codebase, i.e. removing code segments that have been around to allow
> > support for older versions. The overall goal should be to improve the
> Flink
> > codebase in my opinion. Considering what David said in the old thread
> about
> > Hadoop users usually lacking behind with version upgrades [1], would we
> do
> > this version bump in two phases, i.e. adding some deprecation notes and
> > doing the actual cleanup later on? I think Gabor has a point with it not
> > being really mentioned anywhere in the docs (the only location I could
> find
> > in the docs about Hadoop version is [2]). In this sense, the support for
> > older Hadoop versions was kind of implicit: We're talking about compiling
> > Flink with Hadoop 2.8.5 but also mention older Hadoop versions which
> leaves
> > room for interpretation.
> >
> > Additionally, having code that hasn't been touch for a while increases
> the
> > risk of it
> >
> > Matthias
> >
> > [1] https://lists.apache.org/thread/w7www13tossxrxo1mttgb68v81rf6fks
> > [2]
> >
> https://nightlies.a1pache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions
> > <
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#supported-hadoop-versions
> >
> >
> > On Fri, Oct 21, 2022 at 4:13 AM Xintong Song <tonysong...@gmail.com>
> > wrote:
> >
> >> I believe there are some reflection based approaches in the `flink-yarn`
> >> module, for supporting outdated APIs in early Hadoop versions.
> >>
> >> I haven't done a thorough check, and these are what I get.
> >> - AMRMClientAsyncReflector
> >> - ApplicationSubmissionContextReflector
> >> - ContainerRequestReflector
> >> - RegisterApplicationMasterResponseReflector
> >> - ResourceInformationReflector
> >>
> >> Are we removing these as well? If yes, then Flink can no longer work
> with
> >> the old hadoop versions. (That's how I understand "bumping the minimal
> >> supported hadoop version".) I personally am not super eager to get rid
> of
> >> theses, because the relevant parts of codes are no longer frequently
> >> changing, thus the maintenance overhead is low.
> >>
> >> Best,
> >>
> >> Xintong
> >>
> >>
> >>
> >> On Thu, Oct 20, 2022 at 8:00 PM Yang Wang <danrtsey...@gmail.com>
> wrote:
> >>
> >> > Given that we do not bundle any hadoop classes in the Flink binary, do
> >> you
> >> > mean simply bump the hadoop version in the parent pom?
> >> > If it is, why do not we use the latest stable hadoop version 3.3.4? It
> >> > seems that our cron build has verified that hadoop3 could work.
> >> >
> >> > Best,
> >> > Yang
> >> >
> >> > David Morávek <david.mora...@gmail.com> 于2022年10月19日周三 16:29写道:
> >> >
> >> > > +1; anything below 2.10.x seems to be EOL
> >> > >
> >> > > Best,
> >> > > D.
> >> > >
> >> > > On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi <
> >> > balassi.mar...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > Hi Martjin,
> >> > > >
> >> > > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to
> >> > > > implement the bump?
> >> > > >
> >> > > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi <
> >> > gabor.g.somo...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Hi Martin,
> >> > > > >
> >> > > > > Thanks for bringing this up! Lately I was thinking about to bump
> >> the
> >> > > > hadoop
> >> > > > > version to at least 2.6.1 to clean up issues like this:
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
> >> > > > >
> >> > > > > All in all +1 from my perspective.
> >> > > > >
> >> > > > > Just a question here. Are we stating the minimum Hadoop version
> >> for
> >> > > users
> >> > > > > somewhere in the doc or they need to find it out from source
> code
> >> > like
> >> > > > > this?
> >> > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
> >> > > > >
> >> > > > > BR,
> >> > > > > G
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser <
> >> > > martijnvis...@apache.org>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Hi everyone,
> >> > > > > >
> >> > > > > > Little over a year ago a discussion thread was opened on
> >> changing
> >> > the
> >> > > > > > minimal supported version of Hadoop and bringing that to
> 2.8.5.
> >> [1]
> >> > > In
> >> > > > > this
> >> > > > > > discussion thread, I would like to propose to bring that
> minimal
> >> > > > > supported
> >> > > > > > version of Hadoop to 2.10.2.
> >> > > > > >
> >> > > > > > Hadoop 2.8.5 is vulnerable for multiple CVEs which are
> >> classified
> >> > as
> >> > > > > > Critical. [2] [3]. While Flink is not directly impacted by
> >> those,
> >> > we
> >> > > do
> >> > > > > see
> >> > > > > > vulnerability scanners flag Flink as being vulnerable. We
> could
> >> > > easily
> >> > > > > > mitigate that by bumping the minimal supported version of
> >> Hadoop to
> >> > > > > 2.10.2.
> >> > > > > >
> >> > > > > > I'm looking forward to your opinions on this topic.
> >> > > > > >
> >> > > > > > Best regards,
> >> > > > > >
> >> > > > > > Martijn
> >> > > > > > https://twitter.com/MartijnVisser82
> >> > > > > > https://github.com/MartijnVisser
> >> > > > > >
> >> > > > > > [1]
> >> > https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> >> > > > > > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> >> > > > > > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>

Reply via email to