Do we have more cases of "common Hadoop Utils"?

If yes, does it make sense to create a "flink-hadoop-utils" module with
exactly such classes? It would have an optional dependency on
"flink-shaded-hadoop".

On Wed, Mar 4, 2020 at 9:12 AM Till Rohrmann <trohrm...@apache.org> wrote:

> Hi Sivaprasanna,
>
> we don't upload the source jars for the flink-shaded modules. However you
> can build them yourself and install by cloning the flink-shaded repository
> [1] and then call `mvn package -Dshade-sources`.
>
> [1] https://github.com/apache/flink-shaded
>
> Cheers,
> Till
>
> On Tue, Mar 3, 2020 at 6:29 PM Sivaprasanna <sivaprasanna...@gmail.com>
> wrote:
>
> > BTW, can we leverage flink-shaded-hadoop-2? Reason why I ask, if any
> Flink
> > module is going to use Hadoop in any way, it will most probably include
> > flink-shaded-hadoop-2 as a dependency.
> > However, flink-shaded modules don't have any source files. Is that a
> strict
> > convention that the community follows?
> >
> > -
> > Sivaprasanna
> >
> > On Tue, Mar 3, 2020 at 10:48 PM Sivaprasanna <sivaprasanna...@gmail.com>
> > wrote:
> >
> > > Hi Arvid,
> > >
> > > Thanks for the quick reply. Yes, it actually makes sense to avoid
> Hadoop
> > > dependencies from getting into Flink's core modules but I also wonder
> if
> > it
> > > will be an overkill to add flink-hadoop-fs as a dependency just because
> > we
> > > want to use a utility class from that module.
> > >
> > > -
> > > Sivaprasanna
> > >
> > > On Tue, Mar 3, 2020 at 4:17 PM Arvid Heise <ar...@ververica.com>
> wrote:
> > >
> > >> Hi Sivaprasanna,
> > >>
> > >> we actually want to remove Hadoop from all core modules, so we could
> not
> > >> place it in some very common place like flink-core.
> > >>
> > >> But I think the module flink-hadoop-fs could be a fitting place.
> > >>
> > >> On Tue, Mar 3, 2020 at 11:25 AM Sivaprasanna <
> sivaprasanna...@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Hi
> > >> >
> > >> > The flink-sequence-file module has a class named
> > >> > SerializableHadoopConfiguration[1] which is nothing but a wrapper
> > class
> > >> for
> > >> > Hadoop Configuration. I believe this class can be moved to a common
> > >> module
> > >> > since this is not necessarily tightly coupled with sequence-file
> > module,
> > >> > and also because it can be used by many other modules, for ex.
> > >> > flink-compress. Thoughts?
> > >> >
> > >> > -
> > >> > Sivaprasanna
> > >> >
> > >>
> > >
> >
>

Reply via email to