Hi all, For your information, we have document the dependencies detailed information [1]. I think it's a lot clearer than before, but it's worse than presto and spark (they avoid or have built-in hive dependency).
I thought about Stephan's suggestion: - The hive/lib has 200+ jars, but we only need hive-exec.jar or plus two or three jars, if so many jars are introduced, maybe will there be a big conflict. - And hive/lib is not available on every machine. We need to upload so many jars. - A separate classloader maybe hard to work too, our flink-connector-hive need hive jars, we may need to deal with flink-connector-hive jar spacial too. CC: Rui Li I think the best system to integrate with hive is presto, which only connects hive metastore through thrift protocol. But I understand that it costs a lot to rewrite the code. [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/table/hive/#dependencies Best, Jingsong Lee On Tue, Feb 4, 2020 at 1:44 AM Stephan Ewen <se...@apache.org> wrote: > We have had much trouble in the past from "too deep too custom" > integrations that everyone got out of the box, i.e., Hadoop. > Flink has has such a broad spectrum of use cases, if we have custom build > for every other framework in that spectrum, we'll be in trouble. > > So I would also be -1 for custom builds. > > Couldn't we do something similar as we started doing for Hadoop? Moving > away from convenience downloads to allowing users to "export" their setup > for Flink? > > - We can have a "hive module (loader)" in flink/lib by default > - The module loader would look for an environment variable like > "HIVE_CLASSPATH" and load these classes (ideally in a separate > classloader). > - The loader can search for certain classes and instantiate catalog / > functions / etc. when finding them instantiates the hive module referencing > them > - That way, we use exactly what users have installed, without needing to > build our own bundles. > > Could that work? > > Best, > Stephan > > > On Wed, Dec 18, 2019 at 9:43 AM Till Rohrmann <trohrm...@apache.org> > wrote: > > > Couldn't it simply be documented which jars are in the convenience jars > > which are pre built and can be downloaded from the website? Then people > who > > need a custom version know which jars they need to provide to Flink? > > > > Cheers, > > Till > > > > On Tue, Dec 17, 2019 at 6:49 PM Bowen Li <bowenl...@gmail.com> wrote: > > > > > I'm not sure providing an uber jar would be possible. > > > > > > Different from kafka and elasticsearch connector who have dependencies > > for > > > a specific kafka/elastic version, or the kafka universal connector that > > > provides good compatibilities, hive connector needs to deal with hive > > jars > > > in all 1.x, 2.x, 3.x versions (let alone all the HDP/CDH distributions) > > > with incompatibility even between minor versions, different versioned > > > hadoop and other extra dependency jars for each hive version. > > > > > > Besides, users usually need to be able to easily see which individual > > jars > > > are required, which is invisible from an uber jar. Hive users already > > have > > > their hive deployments. They usually have to use their own hive jars > > > because, unlike hive jars on mvn, their own jars contain changes > in-house > > > or from vendors. They need to easily tell which jars Flink requires for > > > corresponding open sourced hive version to their own hive deployment, > and > > > copy in-hosue jars over from hive deployments as replacements. > > > > > > Providing a script to download all the individual jars for a specified > > hive > > > version can be an alternative. > > > > > > The goal is we need to provide a *product*, not a technology, to make > it > > > less hassle for Hive users. Afterall, it's Flink embracing Hive > community > > > and ecosystem, not the other way around. I'd argue Hive connector can > be > > > treat differently because its community/ecosystem/userbase is much > larger > > > than the other connectors, and it's way more important than other > > > connectors to Flink on the mission of becoming a batch/streaming > unified > > > engine and get Flink more widely adopted. > > > > > > > > > On Sun, Dec 15, 2019 at 10:03 PM Danny Chan <yuzhao....@gmail.com> > > wrote: > > > > > > > Also -1 on separate builds. > > > > > > > > After referencing some other BigData engines for distribution[1], i > > > didn't > > > > find strong needs to publish a separate build > > > > for just a separate Hive version, indeed there are builds for > different > > > > Hadoop version. > > > > > > > > Just like Seth and Aljoscha said, we could push a > > > > flink-hive-version-uber.jar to use as a lib of SQL-CLI or other use > > > cases. > > > > > > > > [1] https://spark.apache.org/downloads.html > > > > [2] > > > https://www.elastic.co/guide/en/elasticsearch/hadoop/current/hive.html > > > > > > > > Best, > > > > Danny Chan > > > > 在 2019年12月14日 +0800 AM3:03，dev@flink.apache.org，写道： > > > > > > > > > > > > > > > > > > > > https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html#dependencies > > > > > > > > > > -- Best, Jingsong Lee
