Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-26 Thread Martijn Visser
For those that are interested, I have a draft PR open for this https://github.com/apache/flink/pull/21128 - Feel free to have a look. I'm not sure yet why the Flink CI fails, since these tests are passing locally. Thanks, Martijn On Tue, Oct 25, 2022 at 10:21 AM Matthias Pohl wrote: > > Addit

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-25 Thread Matthias Pohl
> Additionally, having code that hasn't been touch for a while increases the risk of it Sorry about this incomplete confusing sentence. I was about to remove it when accidentally pushing the shortcut for sending the message out to the mailing list. On Tue, Oct 25, 2022 at 10:18 AM Matthias Pohl w

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-25 Thread Matthias Pohl
I guess upgrading the minimal version should also mean cleaning up the codebase, i.e. removing code segments that have been around to allow support for older versions. The overall goal should be to improve the Flink codebase in my opinion. Considering what David said in the old thread about Hadoop

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-20 Thread Xintong Song
I believe there are some reflection based approaches in the `flink-yarn` module, for supporting outdated APIs in early Hadoop versions. I haven't done a thorough check, and these are what I get. - AMRMClientAsyncReflector - ApplicationSubmissionContextReflector - ContainerRequestReflector - Regist

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-20 Thread Yang Wang
Given that we do not bundle any hadoop classes in the Flink binary, do you mean simply bump the hadoop version in the parent pom? If it is, why do not we use the latest stable hadoop version 3.3.4? It seems that our cron build has verified that hadoop3 could work. Best, Yang David Morávek 于2022年

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-19 Thread David Morávek
+1; anything below 2.10.x seems to be EOL Best, D. On Mon, Oct 17, 2022 at 10:48 AM Márton Balassi wrote: > Hi Martjin, > > +1 for 2.10.2. Do you expect to have bandwidth in the near term to > implement the bump? > > On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi > wrote: > > > Hi Martin, > > >

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-17 Thread Márton Balassi
Hi Martjin, +1 for 2.10.2. Do you expect to have bandwidth in the near term to implement the bump? On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi wrote: > Hi Martin, > > Thanks for bringing this up! Lately I was thinking about to bump the hadoop > version to at least 2.6.1 to clean up issues like

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-05 Thread Gabor Somogyi
Hi Martin, Thanks for bringing this up! Lately I was thinking about to bump the hadoop version to at least 2.6.1 to clean up issues like this: https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util

[DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

2022-10-04 Thread Martijn Visser
Hi everyone, Little over a year ago a discussion thread was opened on changing the minimal supported version of Hadoop and bringing that to 2.8.5. [1] In this discussion thread, I would like to propose to bring that minimal supported version of Hadoop to 2.10.2. Hadoop 2.8.5 is vulnerable for mul