Hello,
I think that's a good idea. I suppose that most corporate users use a vendor
version and already compile their flink version with vendor repos anyway.
Arnaud
PS - FYI, for CDH6 (based on hadoop3), I've managed to build a 1.10 version by
modifying pom.xml files and using "hidden" Cloudera package :
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-shaded-hadoop-3</artifactId>
<version>3.1.1.7.0.3.0-79-7.0</version>
<optional>true</optional>
</dependency>
(same with flink-shaded-hadoop-3-uber)
On 2020/04/21 14:37:33, Robert Metzger <r...@apache.org> wrote:
> Hi all,>
>
> for the upcoming 1.11 release, I started looking into adding support for>
> Hadoop 3[1] for Flink. I have explored a little bit already into adding a>
> shaded hadoop 3 into "flink-shaded", and some mechanisms for switching>
> between Hadoop 2 and 3 dependencies in the Flink build.>
>
> However, Chesnay made me aware that we could also go a different route: We>
> let Flink depend on vanilla Hadoop dependencies and stop providing shaded>
> fat jars for Hadoop through "flink-shaded".>
>
> Why?>
> - Maintaining properly shaded Hadoop fat jars is a lot of work (we have>
> insufficient test coverage for all kinds of Hadoop features)>
> - For Hadoop 2, there are already some known and unresolved issues with our>
> shaded jars that we didn't manage to fix>
>
> Users will have to use Flink with Hadoop by relying on vanilla or>
> vendor-provided Hadoop dependencies.>
>
> What do you think?>
>
> Best,>
> Robert>
>
> [1] https://issues.apache.org/jira/browse/FLINK-11086>
>
________________________________
L'int?grit? de ce message n'?tant pas assur?e sur internet, la soci?t?
exp?ditrice ne peut ?tre tenue responsable de son contenu ni de ses pi?ces
jointes. Toute utilisation ou diffusion non autoris?e est interdite. Si vous
n'?tes pas destinataire de ce message, merci de le d?truire et d'avertir
l'exp?diteur.
The integrity of this message cannot be guaranteed on the Internet. The company
that sent this message cannot therefore be held liable for its content nor
attachments. Any unauthorized use or dissemination is prohibited. If you are
not the intended recipient of this message, then please delete it and notify
the sender.
