+1 to add light-weighted formats into the lib On Fri, Jun 5, 2020 at 3:28 PM Leonard Xu <xbjt...@gmail.com> wrote:
> +1 for Jingsong’s proposal to put flink-csv, flink-json and flink-avro > under lib/ directory. > I have heard many SQL users(most of newbies) complaint the out-of-box > experience in mail list. > > Best, > Leonard Xu > > > > 在 2020年6月5日，14:39，Benchao Li <libenc...@gmail.com> 写道： > > > > +1 to include them for sql-client by default; > > +0 to put into lib and exposed to all kinds of jobs, including > DataStream. > > > > Danny Chan <yuzhao....@gmail.com> 于2020年6月5日周五 下午2:31写道： > > > >> +1, at least, we should keep an out of the box SQL-CLI, it’s very poor > >> experience to add such required format jars for SQL users. > >> > >> Best, > >> Danny Chan > >> 在 2020年6月5日 +0800 AM11:14，Jingsong Li <jingsongl...@gmail.com>，写道： > >>> Hi all, > >>> > >>> Considering that 1.11 will be released soon, what about my previous > >>> proposal? Put flink-csv, flink-json and flink-avro under lib. > >>> These three formats are very small and no third party dependence, and > >> they > >>> are widely used by table users. > >>> > >>> Best, > >>> Jingsong Lee > >>> > >>> On Tue, May 12, 2020 at 4:19 PM Jingsong Li <jingsongl...@gmail.com> > >> wrote: > >>> > >>>> Thanks for your discussion. > >>>> > >>>> Sorry to start discussing another thing: > >>>> > >>>> The biggest problem I see is the variety of problems caused by users' > >> lack > >>>> of format dependency. > >>>> As Aljoscha said, these three formats are very small and no third > party > >>>> dependence, and they are widely used by table users. > >>>> Actually, we don't have any other built-in table formats now... In > >> total > >>>> 151K... > >>>> > >>>> 73K flink-avro-1.10.0.jar > >>>> 36K flink-csv-1.10.0.jar > >>>> 42K flink-json-1.10.0.jar > >>>> > >>>> So, Can we just put them into "lib/" or flink-table-uber? > >>>> It not solve all problems and maybe it is independent of "fat" and > >> "slim". > >>>> But also improve usability. > >>>> What do you think? Any objections? > >>>> > >>>> Best, > >>>> Jingsong Lee > >>>> > >>>> On Mon, May 11, 2020 at 5:48 PM Chesnay Schepler <ches...@apache.org> > >>>> wrote: > >>>> > >>>>> One downside would be that we're shipping more stuff when running on > >>>>> YARN for example, since the entire plugins directory is shiped by > >> default. > >>>>> > >>>>> On 17/04/2020 16:38, Stephan Ewen wrote: > >>>>>> @Aljoscha I think that is an interesting line of thinking. the > >> swift-fs > >>>>> may > >>>>>> be rarely enough used to move it to an optional download. > >>>>>> > >>>>>> I would still drop two more thoughts: > >>>>>> > >>>>>> (1) Now that we have plugins support, is there a reason to have a > >>>>> metrics > >>>>>> reporter or file system in /opt instead of /plugins? They don't > >> spoil > >>>>> the > >>>>>> class path any more. > >>>>>> > >>>>>> (2) I can imagine there still being a desire to have a "minimal" > >> docker > >>>>>> file, for users that want to keep the container images as small as > >>>>>> possible, to speed up deployment. It is fine if that would not be > >> the > >>>>>> default, though. > >>>>>> > >>>>>> > >>>>>> On Fri, Apr 17, 2020 at 12:16 PM Aljoscha Krettek < > >> aljos...@apache.org> > >>>>>> wrote: > >>>>>> > >>>>>>> I think having such tools and/or tailor-made distributions can > >> be nice > >>>>>>> but I also think the discussion is missing the main point: The > >> initial > >>>>>>> observation/motivation is that apparently a lot of users (Kurt > >> and I > >>>>>>> talked about this) on the chinese DingTalk support groups, and > >> other > >>>>>>> support channels have problems when first using the SQL client > >> because > >>>>>>> of these missing connectors/formats. For these, having > >> additional tools > >>>>>>> would not solve anything because they would also not take that > >> extra > >>>>>>> step. I think that even tiny friction should be avoided because > >> the > >>>>>>> annoyance from it accumulates of the (hopefully) many users that > >> we > >>>>> want > >>>>>>> to have. > >>>>>>> > >>>>>>> Maybe we should take a step back from discussing the > >> "fat"/"slim" idea > >>>>>>> and instead think about the composition of the current dist. As > >>>>>>> mentioned we have these jars in opt/: > >>>>>>> > >>>>>>> 17M flink-azure-fs-hadoop-1.10.0.jar > >>>>>>> 52K flink-cep-scala_2.11-1.10.0.jar > >>>>>>> 180K flink-cep_2.11-1.10.0.jar > >>>>>>> 746K flink-gelly-scala_2.11-1.10.0.jar > >>>>>>> 626K flink-gelly_2.11-1.10.0.jar > >>>>>>> 512K flink-metrics-datadog-1.10.0.jar > >>>>>>> 159K flink-metrics-graphite-1.10.0.jar > >>>>>>> 1.0M flink-metrics-influxdb-1.10.0.jar > >>>>>>> 102K flink-metrics-prometheus-1.10.0.jar > >>>>>>> 10K flink-metrics-slf4j-1.10.0.jar > >>>>>>> 12K flink-metrics-statsd-1.10.0.jar > >>>>>>> 36M flink-oss-fs-hadoop-1.10.0.jar > >>>>>>> 28M flink-python_2.11-1.10.0.jar > >>>>>>> 22K flink-queryable-state-runtime_2.11-1.10.0.jar > >>>>>>> 18M flink-s3-fs-hadoop-1.10.0.jar > >>>>>>> 31M flink-s3-fs-presto-1.10.0.jar > >>>>>>> 196K flink-shaded-netty-tcnative-dynamic-2.0.25.Final-9.0.jar > >>>>>>> 518K flink-sql-client_2.11-1.10.0.jar > >>>>>>> 99K flink-state-processor-api_2.11-1.10.0.jar > >>>>>>> 25M flink-swift-fs-hadoop-1.10.0.jar > >>>>>>> 160M opt > >>>>>>> > >>>>>>> The "filesystem" connectors ar ethe heavy hitters, there. > >>>>>>> > >>>>>>> I downloaded most of the SQL connectors/formats and this is what > >> I got: > >>>>>>> > >>>>>>> 73K flink-avro-1.10.0.jar > >>>>>>> 36K flink-csv-1.10.0.jar > >>>>>>> 55K flink-hbase_2.11-1.10.0.jar > >>>>>>> 88K flink-jdbc_2.11-1.10.0.jar > >>>>>>> 42K flink-json-1.10.0.jar > >>>>>>> 20M flink-sql-connector-elasticsearch6_2.11-1.10.0.jar > >>>>>>> 2.8M flink-sql-connector-kafka_2.11-1.10.0.jar > >>>>>>> 24M sql-connectors-formats > >>>>>>> > >>>>>>> We could just add these to the Flink distribution without > >> blowing it up > >>>>>>> by much. We could drop any of the existing "filesystem" > >> connectors from > >>>>>>> opt and add the SQL connectors/formats and not change the size > >> of Flink > >>>>>>> dist. So maybe we should do that instead? > >>>>>>> > >>>>>>> We would need some tooling for the sql-client shell script to > >> pick-up > >>>>>>> the connectors/formats up from opt/ because we don't want to add > >> them > >>>>> to > >>>>>>> lib/. We're already doing that for finding the flink-sql-client > >> jar, > >>>>>>> which is also not in lib/. > >>>>>>> > >>>>>>> What do you think? > >>>>>>> > >>>>>>> Best, > >>>>>>> Aljoscha > >>>>>>> > >>>>>>> On 17.04.20 05:22, Jark Wu wrote: > >>>>>>>> Hi, > >>>>>>>> > >>>>>>>> I like the idea of web tool to assemble fat distribution. And > >> the > >>>>>>>> https://code.quarkus.io/ looks very nice. > >>>>>>>> All the users need to do is just select what he/she need (I > >> think this > >>>>>>> step > >>>>>>>> can't be omitted anyway). > >>>>>>>> We can also provide a default fat distribution on the web which > >>>>> default > >>>>>>>> selects some popular connectors. > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Jark > >>>>>>>> > >>>>>>>> On Fri, 17 Apr 2020 at 02:29, Rafi Aroch <rafi.ar...@gmail.com > >>> > >>>>> wrote: > >>>>>>>> > >>>>>>>>> As a reference for a nice first-experience I had, take a > >> look at > >>>>>>>>> https://code.quarkus.io/ > >>>>>>>>> You reach this page after you click "Start Coding" at the > >> project > >>>>>>> homepage. > >>>>>>>>> Rafi > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Thu, Apr 16, 2020 at 6:53 PM Kurt Young <ykt...@gmail.com> > >> wrote: > >>>>>>>>> > >>>>>>>>>> I'm not saying pre-bundle some jars will make this problem > >> go away, > >>>>> and > >>>>>>>>>> you're right that only hides the problem for > >>>>>>>>>> some users. But what if this solution can hide the problem > >> for 90% > >>>>>>> users? > >>>>>>>>>> Would't that be good enough for us to try? > >>>>>>>>>> > >>>>>>>>>> Regarding to would users following instructions really be > >> such a big > >>>>>>>>>> problem? > >>>>>>>>>> I'm afraid yes. Otherwise I won't answer such questions > >> for at > >>>>> least a > >>>>>>>>>> dozen times and I won't see such questions coming > >>>>>>>>>> up from time to time. During some periods, I even saw such > >> questions > >>>>>>>>> every > >>>>>>>>>> day. > >>>>>>>>>> > >>>>>>>>>> Best, > >>>>>>>>>> Kurt > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Apr 16, 2020 at 11:21 PM Chesnay Schepler < > >>>>> ches...@apache.org> > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> The problem with having a distribution with "popular" > >> stuff is > >>>>> that it > >>>>>>>>>>> doesn't really *solve* a problem, it just hides it for > >> users who > >>>>> fall > >>>>>>>>>>> into these particular use-cases. > >>>>>>>>>>> Move out of it and you once again run into exact same > >> problems > >>>>>>>>> out-lined. > >>>>>>>>>>> This is exactly why I like the tooling approach; you > >> have to deal > >>>>> with > >>>>>>>>> it > >>>>>>>>>>> from the start and transitioning to a custom use-case is > >> easier. > >>>>>>>>>>> > >>>>>>>>>>> Would users following instructions really be such a big > >> problem? > >>>>>>>>>>> I would expect that users generally know *what *they > >> need, just not > >>>>>>>>>>> necessarily how it is assembled correctly (where do get > >> which jar, > >>>>>>>>> which > >>>>>>>>>>> directory to put it in). > >>>>>>>>>>> It seems like these are exactly the problem this would > >> solve? > >>>>>>>>>>> I just don't see how moving a jar corresponding to some > >> feature > >>>>> from > >>>>>>>>> opt > >>>>>>>>>>> to some directory (lib/plugins) is less error-prone than > >> just > >>>>>>> selecting > >>>>>>>>>> the > >>>>>>>>>>> feature and having the tool handle the rest. > >>>>>>>>>>> > >>>>>>>>>>> As for re-distributions, it depends on the form that the > >> tool would > >>>>>>>>> take. > >>>>>>>>>>> It could be an application that runs locally and works > >> against > >>>>> maven > >>>>>>>>>>> central (note: not necessarily *using* maven); this > >> should would > >>>>> work > >>>>>>>>> in > >>>>>>>>>>> China, no? > >>>>>>>>>>> > >>>>>>>>>>> A web tool would of course be fancy, but I don't know > >> how feasible > >>>>>>> this > >>>>>>>>>> is > >>>>>>>>>>> with the ASF infrastructure. > >>>>>>>>>>> You wouldn't be able to mirror the distribution, so the > >> load can't > >>>>> be > >>>>>>>>>>> distributed. I doubt INFRA would like this. > >>>>>>>>>>> > >>>>>>>>>>> Note that third-parties could also start distributing > >> use-case > >>>>>>> oriented > >>>>>>>>>>> distributions, which would be perfectly fine as far as > >> I'm > >>>>> concerned. > >>>>>>>>>>> > >>>>>>>>>>> On 16/04/2020 16:57, Kurt Young wrote: > >>>>>>>>>>> > >>>>>>>>>>> I'm not so sure about the web tool solution though. The > >> concern I > >>>>> have > >>>>>>>>>> for > >>>>>>>>>>> this approach is the final generated > >>>>>>>>>>> distribution is kind of non-deterministic. We might > >> generate too > >>>>> many > >>>>>>>>>>> different combinations when user trying to > >>>>>>>>>>> package different types of connector, format, and even > >> maybe hadoop > >>>>>>>>>>> releases. As far as I can tell, most open > >>>>>>>>>>> source projects and apache projects will only release > >> some > >>>>>>>>>>> pre-defined distributions, which most users are already > >>>>>>>>>>> familiar with, thus hard to change IMO. And I also have > >> went > >>>>> through > >>>>>>> in > >>>>>>>>>>> some cases, users will try to re-distribute > >>>>>>>>>>> the release package, because of the unstable network of > >> apache > >>>>> website > >>>>>>>>>> from > >>>>>>>>>>> China. In web tool solution, I don't > >>>>>>>>>>> think this kind of re-distribution would be possible > >> anymore. > >>>>>>>>>>> > >>>>>>>>>>> In the meantime, I also have a concern that we will fall > >> back into > >>>>> our > >>>>>>>>>> trap > >>>>>>>>>>> again if we try to offer this smart & flexible > >>>>>>>>>>> solution. Because it needs users to cooperate with such > >> mechanism. > >>>>>>> It's > >>>>>>>>>>> exactly the situation what we currently fell > >>>>>>>>>>> into: > >>>>>>>>>>> 1. We offered a smart solution. > >>>>>>>>>>> 2. We hope users will follow the correct instructions. > >>>>>>>>>>> 3. Everything will work as expected if users followed > >> the right > >>>>>>>>>>> instructions. > >>>>>>>>>>> > >>>>>>>>>>> In reality, I suspect not all users will do the second > >> step > >>>>> correctly. > >>>>>>>>>> And > >>>>>>>>>>> for new users who only trying to have a quick > >>>>>>>>>>> experience with Flink, I would bet most users will do it > >> wrong. > >>>>>>>>>>> > >>>>>>>>>>> So, my proposal would be one of the following 2 options: > >>>>>>>>>>> 1. Provide a slim distribution for advanced product > >> users and > >>>>> provide > >>>>>>> a > >>>>>>>>>>> distribution which will have some popular builtin jars. > >>>>>>>>>>> 2. Only provide a distribution which will have some > >> popular builtin > >>>>>>>>> jars. > >>>>>>>>>>> If we are trying to reduce the distributions we > >> released, I would > >>>>>>>>> prefer > >>>>>>>>>> 2 > >>>>>>>>>>> 1. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Kurt > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Apr 16, 2020 at 9:33 PM Till Rohrmann < > >>>>> trohrm...@apache.org> > >>>>>>> < > >>>>>>>>>> trohrm...@apache.org> wrote: > >>>>>>>>>>> > >>>>>>>>>>> I think what Chesnay and Dawid proposed would be the > >> ideal > >>>>> solution. > >>>>>>>>>>> Ideally, we would also have a nice web tool for the > >> website which > >>>>>>>>>> generates > >>>>>>>>>>> the corresponding distribution for download. > >>>>>>>>>>> > >>>>>>>>>>> To get things started we could start with only > >> supporting to > >>>>>>>>>>> download/creating the "fat" version with the script. The > >> fat > >>>>> version > >>>>>>>>>> would > >>>>>>>>>>> then consist of the slim distribution and whatever we > >> deem > >>>>> important > >>>>>>>>> for > >>>>>>>>>>> new users to get started. > >>>>>>>>>>> > >>>>>>>>>>> Cheers, > >>>>>>>>>>> Till > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Apr 16, 2020 at 11:33 AM Dawid Wysakowicz < > >>>>>>>>>> dwysakow...@apache.org> <dwysakow...@apache.org> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Hi all, > >>>>>>>>>>> > >>>>>>>>>>> Few points from my side: > >>>>>>>>>>> > >>>>>>>>>>> 1. I like the idea of simplifying the experience for > >> first time > >>>>> users. > >>>>>>>>>>> As for production use cases I share Jark's opinion that > >> in this > >>>>> case I > >>>>>>>>>>> would expect users to combine their distribution > >> manually. I think > >>>>> in > >>>>>>>>>>> such scenarios it is important to understand > >> interconnections. > >>>>>>>>>>> Personally I'd expect the slimmest possible distribution > >> that I can > >>>>>>>>>>> extend further with what I need in my production > >> scenario. > >>>>>>>>>>> > >>>>>>>>>>> 2. I think there is also the problem that the matrix of > >> possible > >>>>>>>>>>> combinations that can be useful is already big. Do we > >> want to have > >>>>> a > >>>>>>>>>>> distribution for: > >>>>>>>>>>> > >>>>>>>>>>> SQL users: which connectors should we include? should we > >>>>> include > >>>>>>>>>>> hive? which other catalog? > >>>>>>>>>>> > >>>>>>>>>>> DataStream users: which connectors should we include? > >>>>>>>>>>> > >>>>>>>>>>> For both of the above should we include yarn/kubernetes? > >>>>>>>>>>> > >>>>>>>>>>> I would opt for providing only the "slim" distribution > >> as a release > >>>>>>>>>>> artifact. > >>>>>>>>>>> > >>>>>>>>>>> 3. However, as I said I think its worth investigating > >> how we can > >>>>>>>>> improve > >>>>>>>>>>> users experience. What do you think of providing a tool, > >> could be > >>>>> e.g. > >>>>>>>>> a > >>>>>>>>>>> shell script that constructs a distribution based on > >> users choice. > >>>>> I > >>>>>>>>>>> think that was also what Chesnay mentioned as "tooling to > >>>>>>>>>>> assemble custom distributions" In the end how I see the > >> difference > >>>>>>>>>>> between a slim and fat distribution is which jars do we > >> put into > >>>>> the > >>>>>>>>>>> lib, right? It could have a few "screens". > >>>>>>>>>>> > >>>>>>>>>>> 1. Which API are you interested in: > >>>>>>>>>>> a. SQL API > >>>>>>>>>>> b. DataStream API > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> 2. [SQL] Which connectors do you want to use? > >> [multichoice]: > >>>>>>>>>>> a. Kafka > >>>>>>>>>>> b. Elasticsearch > >>>>>>>>>>> ... > >>>>>>>>>>> > >>>>>>>>>>> 3. [SQL] Which catalog you want to use? > >>>>>>>>>>> > >>>>>>>>>>> ... > >>>>>>>>>>> > >>>>>>>>>>> Such a tool would download all the dependencies from > >> maven and put > >>>>>>> them > >>>>>>>>>>> into the correct folder. In the future we can extend it > >> with > >>>>>>> additional > >>>>>>>>>>> rules e.g. kafka-0.9 cannot be chosen at the same time > >> with > >>>>>>>>>>> kafka-universal etc. > >>>>>>>>>>> > >>>>>>>>>>> The benefit of it would be that the distribution that we > >> release > >>>>> could > >>>>>>>>>>> remain "slim" or we could even make it slimmer. I might > >> be missing > >>>>>>>>>>> something here though. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> > >>>>>>>>>>> Dawdi > >>>>>>>>>>> > >>>>>>>>>>> On 16/04/2020 11:02, Aljoscha Krettek wrote: > >>>>>>>>>>> > >>>>>>>>>>> I want to reinforce my opinion from earlier: This is > >> about > >>>>> improving > >>>>>>>>>>> the situation both for first-time users and for > >> experienced users > >>>>> that > >>>>>>>>>>> want to use a Flink dist in production. The current > >> Flink dist is > >>>>> too > >>>>>>>>>>> "thin" for first-time SQL users and it is too "fat" for > >> production > >>>>>>>>>>> users, that is where serving no-one properly with the > >> current > >>>>>>>>>>> middle-ground. That's why I think introducing those > >> specialized > >>>>>>>>>>> "spins" of Flink dist would be good. > >>>>>>>>>>> > >>>>>>>>>>> By the way, at some point in the future production users > >> might not > >>>>>>>>>>> even need to get a Flink dist anymore. They should be > >> able to have > >>>>>>>>>>> Flink as a dependency of their project (including the > >> runtime) and > >>>>>>>>>>> then build an image from this for Kubernetes or a fat > >> jar for YARN. > >>>>>>>>>>> > >>>>>>>>>>> Aljoscha > >>>>>>>>>>> > >>>>>>>>>>> On 15.04.20 18:14, wenlong.lwl wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi all, > >>>>>>>>>>> > >>>>>>>>>>> Regarding slim and fat distributions, I think different > >> kinds of > >>>>> jobs > >>>>>>>>>>> may > >>>>>>>>>>> prefer different type of distribution: > >>>>>>>>>>> > >>>>>>>>>>> For DataStream job, I think we may not like fat > >> distribution > >>>>>>>>>>> > >>>>>>>>>>> containing > >>>>>>>>>>> > >>>>>>>>>>> connectors because user would always need to depend on > >> the > >>>>> connector > >>>>>>>>>>> > >>>>>>>>>>> in > >>>>>>>>>>> > >>>>>>>>>>> user code, it is easy to include the connector jar in > >> the user lib. > >>>>>>>>>>> > >>>>>>>>>>> Less > >>>>>>>>>>> > >>>>>>>>>>> jar in lib means less class conflicts and problems. > >>>>>>>>>>> > >>>>>>>>>>> For SQL job, I think we are trying to encourage user to > >> user pure > >>>>>>>>>>> sql(DDL + > >>>>>>>>>>> DML) to construct their job, In order to improve user > >> experience, > >>>>> It > >>>>>>>>>>> may be > >>>>>>>>>>> important for flink, not only providing as many > >> connector jar in > >>>>>>>>>>> distribution as possible especially the connector and > >> format we > >>>>> have > >>>>>>>>>>> well > >>>>>>>>>>> documented, but also providing an mechanism to load > >> connectors > >>>>>>>>>>> according > >>>>>>>>>>> to the DDLs, > >>>>>>>>>>> > >>>>>>>>>>> So I think it could be good to place connector/format > >> jars in some > >>>>>>>>>>> dir like > >>>>>>>>>>> opt/connector which would not affect jobs by default, and > >>>>> introduce a > >>>>>>>>>>> mechanism of dynamic discovery for SQL. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Wenlong > >>>>>>>>>>> > >>>>>>>>>>> On Wed, 15 Apr 2020 at 22:46, Jingsong Li < > >> jingsongl...@gmail.com> > >>>>> < > >>>>>>>>>> jingsongl...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> I am thinking both "improve first experience" and > >> "improve > >>>>> production > >>>>>>>>>>> experience". > >>>>>>>>>>> > >>>>>>>>>>> I'm thinking about what's the common mode of Flink? > >>>>>>>>>>> Streaming job use Kafka? Batch job use Hive? > >>>>>>>>>>> > >>>>>>>>>>> Hive 1.2.1 dependencies can be compatible with most of > >> Hive server > >>>>>>>>>>> versions. So Spark and Presto have built-in Hive 1.2.1 > >> dependency. > >>>>>>>>>>> Flink is currently mainly used for streaming, so let's > >> not talk > >>>>>>>>>>> about hive. > >>>>>>>>>>> > >>>>>>>>>>> For streaming jobs, first of all, the jobs in my mind is > >> (related > >>>>> to > >>>>>>>>>>> connectors): > >>>>>>>>>>> - ETL jobs: Kafka -> Kafka > >>>>>>>>>>> - Join jobs: Kafka -> DimJDBC -> Kafka > >>>>>>>>>>> - Aggregation jobs: Kafka -> JDBCSink > >>>>>>>>>>> So Kafka and JDBC are probably the most commonly used. > >> Of course, > >>>>>>>>>>> > >>>>>>>>>>> also > >>>>>>>>>>> > >>>>>>>>>>> includes CSV, JSON's formats. > >>>>>>>>>>> So when we provide such a fat distribution: > >>>>>>>>>>> - With CSV, JSON. > >>>>>>>>>>> - With flink-kafka-universal and kafka dependencies. > >>>>>>>>>>> - With flink-jdbc. > >>>>>>>>>>> Using this fat distribution, most users can run their > >> jobs well. > >>>>>>>>>>> > >>>>>>>>>>> (jdbc > >>>>>>>>>>> > >>>>>>>>>>> driver jar required, but this is very natural to do) > >>>>>>>>>>> Can these dependencies lead to kinds of conflicts? Only > >> Kafka may > >>>>>>>>>>> > >>>>>>>>>>> have > >>>>>>>>>>> > >>>>>>>>>>> conflicts, but if our goal is to use kafka-universal to > >> support all > >>>>>>>>>>> Kafka > >>>>>>>>>>> versions, it is hopeful to target the vast majority of > >> users. > >>>>>>>>>>> > >>>>>>>>>>> We don't want to plug all jars into the fat > >> distribution. Only need > >>>>>>>>>>> less > >>>>>>>>>>> conflict and common. of course, it is a matter of > >> consideration to > >>>>>>>>>>> > >>>>>>>>>>> put > >>>>>>>>>>> > >>>>>>>>>>> which jar into fat distribution. > >>>>>>>>>>> We have the opportunity to facilitate the majority of > >> users, but > >>>>>>>>>>> also left > >>>>>>>>>>> opportunities for customization. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Jingsong Lee > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Apr 15, 2020 at 10:09 PM Jark Wu < > >> imj...@gmail.com> < > >>>>>>>>>> imj...@gmail.com> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> I think we should first reach an consensus on "what > >> problem do we > >>>>>>>>>>> want to > >>>>>>>>>>> solve?" > >>>>>>>>>>> (1) improve first experience? or (2) improve production > >> experience? > >>>>>>>>>>> > >>>>>>>>>>> As far as I can see, with the above discussion, I think > >> what we > >>>>>>>>>>> want to > >>>>>>>>>>> solve is the "first experience". > >>>>>>>>>>> And I think the slim jar is still the best distribution > >> for > >>>>>>>>>>> production, > >>>>>>>>>>> because it's easier to assembling jars > >>>>>>>>>>> than excluding jars and can avoid potential class > >> conflicts. > >>>>>>>>>>> > >>>>>>>>>>> If we want to improve "first experience", I think it > >> make sense to > >>>>>>>>>>> have a > >>>>>>>>>>> fat distribution to give users a more smooth first > >> experience. > >>>>>>>>>>> But I would like to call it "playground distribution" or > >> something > >>>>>>>>>>> like > >>>>>>>>>>> that to explicitly differ from the "slim > >> production-purpose > >>>>>>>>>>> > >>>>>>>>>>> distribution". > >>>>>>>>>>> > >>>>>>>>>>> The "playground distribution" can contains some widely > >> used jars, > >>>>>>>>>>> > >>>>>>>>>>> like > >>>>>>>>>>> > >>>>>>>>>>> universal-kafka-sql-connector, > >> elasticsearch7-sql-connector, avro, > >>>>>>>>>>> json, > >>>>>>>>>>> csv, etc.. > >>>>>>>>>>> Even we can provide a playground docker which may > >> contain the fat > >>>>>>>>>>> distribution, python3, and hive. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Jark > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Wed, 15 Apr 2020 at 21:47, Chesnay Schepler < > >> ches...@apache.org> > >>>>> < > >>>>>>>>>> ches...@apache.org> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> I don't see a lot of value in having multiple > >> distributions. > >>>>>>>>>>> > >>>>>>>>>>> The simple reality is that no fat distribution we could > >> provide > >>>>>>>>>>> > >>>>>>>>>>> would > >>>>>>>>>>> > >>>>>>>>>>> satisfy all use-cases, so why even try. > >>>>>>>>>>> If users commonly run into issues for certain jars, then > >> maybe > >>>>>>>>>>> > >>>>>>>>>>> those > >>>>>>>>>>> > >>>>>>>>>>> should be added to the current distribution. > >>>>>>>>>>> > >>>>>>>>>>> Personally though I still believe we should only > >> distribute a slim > >>>>>>>>>>> version. I'd rather have users always add required jars > >> to the > >>>>>>>>>>> distribution than only when they go outside our > >> "expected" > >>>>>>>>>>> > >>>>>>>>>>> use-cases. > >>>>>>>>>>> > >>>>>>>>>>> Then we might finally address this issue properly, i.e., > >> tooling to > >>>>>>>>>>> assemble custom distributions and/or better error > >> messages if > >>>>>>>>>>> Flink-provided extensions cannot be found. > >>>>>>>>>>> > >>>>>>>>>>> On 15/04/2020 15:23, Kurt Young wrote: > >>>>>>>>>>> > >>>>>>>>>>> Regarding to the specific solution, I'm not sure about > >> the "fat" > >>>>>>>>>>> > >>>>>>>>>>> and > >>>>>>>>>>> > >>>>>>>>>>> "slim" > >>>>>>>>>>> > >>>>>>>>>>> solution though. I get the idea > >>>>>>>>>>> that we can make the slim one even more lightweight than > >> current > >>>>>>>>>>> distribution, but what about the "fat" > >>>>>>>>>>> one? Do you mean that we would package all connectors > >> and formats > >>>>>>>>>>> > >>>>>>>>>>> into > >>>>>>>>>>> > >>>>>>>>>>> this? I'm not sure if this is > >>>>>>>>>>> feasible. For example, we can't put all versions of > >> kafka and hive > >>>>>>>>>>> connector jars into lib directory, and > >>>>>>>>>>> we also might need hadoop jars when using filesystem > >> connector to > >>>>>>>>>>> > >>>>>>>>>>> access > >>>>>>>>>>> > >>>>>>>>>>> data from HDFS. > >>>>>>>>>>> > >>>>>>>>>>> So my guess would be we might hand-pick some of the most > >>>>>>>>>>> > >>>>>>>>>>> frequently > >>>>>>>>>>> > >>>>>>>>>>> used > >>>>>>>>>>> > >>>>>>>>>>> connectors and formats > >>>>>>>>>>> into our "lib" directory, like kafka, csv, json metioned > >> above, > >>>>>>>>>>> > >>>>>>>>>>> and > >>>>>>>>>>> > >>>>>>>>>>> still > >>>>>>>>>>> > >>>>>>>>>>> leave some other connectors out of it. > >>>>>>>>>>> If this is the case, then why not we just provide this > >>>>>>>>>>> > >>>>>>>>>>> distribution > >>>>>>>>>>> > >>>>>>>>>>> to > >>>>>>>>>>> > >>>>>>>>>>> user? I'm not sure i get the benefit of > >>>>>>>>>>> providing another super "slim" jar (we have to pay some > >> costs to > >>>>>>>>>>> > >>>>>>>>>>> provide > >>>>>>>>>>> > >>>>>>>>>>> another suit of distribution). > >>>>>>>>>>> > >>>>>>>>>>> What do you think? > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Kurt > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Apr 15, 2020 at 7:08 PM Jingsong Li < > >>>>>>>>>>> > >>>>>>>>>>> jingsongl...@gmail.com > >>>>>>>>>>> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Big +1. > >>>>>>>>>>> > >>>>>>>>>>> I like "fat" and "slim". > >>>>>>>>>>> > >>>>>>>>>>> For csv and json, like Jark said, they are quite small > >> and don't > >>>>>>>>>>> > >>>>>>>>>>> have > >>>>>>>>>>> > >>>>>>>>>>> other > >>>>>>>>>>> > >>>>>>>>>>> dependencies. They are important to kafka connector, and > >>>>>>>>>>> > >>>>>>>>>>> important > >>>>>>>>>>> > >>>>>>>>>>> to upcoming file system connector too. > >>>>>>>>>>> So can we move them to both "fat" and "slim"? They're so > >>>>>>>>>>> > >>>>>>>>>>> important, > >>>>>>>>>>> > >>>>>>>>>>> and > >>>>>>>>>>> > >>>>>>>>>>> they're so lightweight. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Jingsong Lee > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Apr 15, 2020 at 4:53 PM godfrey he < > >> godfre...@gmail.com> < > >>>>>>>>>> godfre...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Big +1. > >>>>>>>>>>> This will improve user experience (special for Flink new > >> users). > >>>>>>>>>>> We answered so many questions about "class not found". > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Godfrey > >>>>>>>>>>> > >>>>>>>>>>> Dian Fu <dian0511...@gmail.com> <dian0511...@gmail.com> > >>>>> 于2020年4月15日周三 > >>>>>>>>>> 下午4:30写道： > >>>>>>>>>>> > >>>>>>>>>>> +1 to this proposal. > >>>>>>>>>>> > >>>>>>>>>>> Missing connector jars is also a big problem for PyFlink > >> users. > >>>>>>>>>>> > >>>>>>>>>>> Currently, > >>>>>>>>>>> > >>>>>>>>>>> after a Python user has installed PyFlink using `pip`, > >> he has > >>>>>>>>>>> > >>>>>>>>>>> to > >>>>>>>>>>> > >>>>>>>>>>> manually > >>>>>>>>>>> > >>>>>>>>>>> copy the connector fat jars to the PyFlink installation > >>>>>>>>>>> > >>>>>>>>>>> directory > >>>>>>>>>>> > >>>>>>>>>>> for > >>>>>>>>>>> > >>>>>>>>>>> the > >>>>>>>>>>> > >>>>>>>>>>> connectors to be used if he wants to run jobs locally. > >> This > >>>>>>>>>>> > >>>>>>>>>>> process > >>>>>>>>>>> > >>>>>>>>>>> is > >>>>>>>>>>> > >>>>>>>>>>> very > >>>>>>>>>>> > >>>>>>>>>>> confuse for users and affects the experience a lot. > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Dian > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> 在 2020年4月15日，下午3:51，Jark Wu <imj...@gmail.com> < > >> imj...@gmail.com> > >>>>> 写道： > >>>>>>>>>>> > >>>>>>>>>>> +1 to the proposal. I also found the "download > >> additional jar" > >>>>>>>>>>> > >>>>>>>>>>> step > >>>>>>>>>>> > >>>>>>>>>>> is > >>>>>>>>>>> > >>>>>>>>>>> really verbose when I prepare webinars. > >>>>>>>>>>> > >>>>>>>>>>> At least, I think the flink-csv and flink-json should in > >> the > >>>>>>>>>>> > >>>>>>>>>>> distribution, > >>>>>>>>>>> > >>>>>>>>>>> they are quite small and don't have other dependencies. > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Jark > >>>>>>>>>>> > >>>>>>>>>>> On Wed, 15 Apr 2020 at 15:44, Jeff Zhang < > >> zjf...@gmail.com> < > >>>>>>>>>> zjf...@gmail.com> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Hi Aljoscha, > >>>>>>>>>>> > >>>>>>>>>>> Big +1 for the fat flink distribution, where do you plan > >> to > >>>>>>>>>>> > >>>>>>>>>>> put > >>>>>>>>>>> > >>>>>>>>>>> these > >>>>>>>>>>> > >>>>>>>>>>> connectors ? opt or lib ? > >>>>>>>>>>> > >>>>>>>>>>> Aljoscha Krettek <aljos...@apache.org> < > >> aljos...@apache.org> > >>>>>>>>>> 于2020年4月15日周三 > >>>>>>>>>>> 下午3:30写道： > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Hi Everyone, > >>>>>>>>>>> > >>>>>>>>>>> I'd like to discuss about releasing a more full-featured > >>>>>>>>>>> > >>>>>>>>>>> Flink > >>>>>>>>>>> > >>>>>>>>>>> distribution. The motivation is that there is friction > >> for > >>>>>>>>>>> > >>>>>>>>>>> SQL/Table > >>>>>>>>>>> > >>>>>>>>>>> API > >>>>>>>>>>> > >>>>>>>>>>> users that want to use Table connectors which are not > >> there > >>>>>>>>>>> > >>>>>>>>>>> in > >>>>>>>>>>> > >>>>>>>>>>> the > >>>>>>>>>>> > >>>>>>>>>>> current Flink Distribution. For these users the workflow > >> is > >>>>>>>>>>> > >>>>>>>>>>> currently > >>>>>>>>>>> > >>>>>>>>>>> roughly: > >>>>>>>>>>> > >>>>>>>>>>> - download Flink dist > >>>>>>>>>>> - configure csv/Kafka/json connectors per configuration > >>>>>>>>>>> - run SQL client or program > >>>>>>>>>>> - decrypt error message and research the solution > >>>>>>>>>>> - download additional connector jars > >>>>>>>>>>> - program works correctly > >>>>>>>>>>> > >>>>>>>>>>> I realize that this can be made to work but if every SQL > >>>>>>>>>>> > >>>>>>>>>>> user > >>>>>>>>>>> > >>>>>>>>>>> has > >>>>>>>>>>> > >>>>>>>>>>> this > >>>>>>>>>>> > >>>>>>>>>>> as their first experience that doesn't seem good to me. > >>>>>>>>>>> > >>>>>>>>>>> My proposal is to provide two versions of the Flink > >>>>>>>>>>> > >>>>>>>>>>> Distribution > >>>>>>>>>>> > >>>>>>>>>>> in > >>>>>>>>>>> > >>>>>>>>>>> the > >>>>>>>>>>> > >>>>>>>>>>> future: "fat" and "slim" (names to be discussed): > >>>>>>>>>>> > >>>>>>>>>>> - slim would be even trimmer than todays distribution > >>>>>>>>>>> - fat would contain a lot of convenience connectors (yet > >>>>>>>>>>> > >>>>>>>>>>> to > >>>>>>>>>>> > >>>>>>>>>>> be > >>>>>>>>>>> > >>>>>>>>>>> determined which one) > >>>>>>>>>>> > >>>>>>>>>>> And yes, I realize that there are already more > >> dimensions of > >>>>>>>>>>> > >>>>>>>>>>> Flink > >>>>>>>>>>> > >>>>>>>>>>> releases (Scala version and Java version). > >>>>>>>>>>> > >>>>>>>>>>> For background, our current Flink dist has these in the > >> opt > >>>>>>>>>>> > >>>>>>>>>>> directory: > >>>>>>>>>>> > >>>>>>>>>>> - flink-azure-fs-hadoop-1.10.0.jar > >>>>>>>>>>> - flink-cep-scala_2.12-1.10.0.jar > >>>>>>>>>>> - flink-cep_2.12-1.10.0.jar > >>>>>>>>>>> - flink-gelly-scala_2.12-1.10.0.jar > >>>>>>>>>>> - flink-gelly_2.12-1.10.0.jar > >>>>>>>>>>> - flink-metrics-datadog-1.10.0.jar > >>>>>>>>>>> - flink-metrics-graphite-1.10.0.jar > >>>>>>>>>>> - flink-metrics-influxdb-1.10.0.jar > >>>>>>>>>>> - flink-metrics-prometheus-1.10.0.jar > >>>>>>>>>>> - flink-metrics-slf4j-1.10.0.jar > >>>>>>>>>>> - flink-metrics-statsd-1.10.0.jar > >>>>>>>>>>> - flink-oss-fs-hadoop-1.10.0.jar > >>>>>>>>>>> - flink-python_2.12-1.10.0.jar > >>>>>>>>>>> - flink-queryable-state-runtime_2.12-1.10.0.jar > >>>>>>>>>>> - flink-s3-fs-hadoop-1.10.0.jar > >>>>>>>>>>> - flink-s3-fs-presto-1.10.0.jar > >>>>>>>>>>> - > >>>>>>>>>>> > >>>>>>>>>>> flink-shaded-netty-tcnative-dynamic-2.0.25.Final-9.0.jar > >>>>>>>>>>> > >>>>>>>>>>> - flink-sql-client_2.12-1.10.0.jar > >>>>>>>>>>> - flink-state-processor-api_2.12-1.10.0.jar > >>>>>>>>>>> - flink-swift-fs-hadoop-1.10.0.jar > >>>>>>>>>>> > >>>>>>>>>>> Current Flink dist is 267M. If we removed everything from > >>>>>>>>>>> > >>>>>>>>>>> opt > >>>>>>>>>>> > >>>>>>>>>>> we > >>>>>>>>>>> > >>>>>>>>>>> would > >>>>>>>>>>> > >>>>>>>>>>> go down to 126M. I would reccomend this, because the > >> large > >>>>>>>>>>> > >>>>>>>>>>> majority > >>>>>>>>>>> > >>>>>>>>>>> of > >>>>>>>>>>> > >>>>>>>>>>> the files in opt are probably unused. > >>>>>>>>>>> > >>>>>>>>>>> What do you think? > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> Aljoscha > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Best Regards > >>>>>>>>>>> > >>>>>>>>>>> Jeff Zhang > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Best, Jingsong Lee > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Best, Jingsong Lee > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>> > >>>>> > >>>>> > >>>> > >>>> -- > >>>> Best, Jingsong Lee > >>>> > >>> > >>> > >>> -- > >>> Best, Jingsong Lee > >> > > > > > > -- > > > > Best, > > Benchao Li > > -- Best regards! Rui Li
