Big +1 from my side. >From my experience, missing connector & format jar is the TOP 1 problem which SQL users will probably run into. Similar questions raised in Flink's Dingtalk group almost every 1 or 2 days. And I have personally answered dozens of such question. Sometimes it's still not enough for users to download the jars and put it in lib directory, they also have to restart their Flink cluster, which is not obvious and the situation will look like very tricky.
Best, Kurt On Wed, Apr 15, 2020 at 3:30 PM Aljoscha Krettek <aljos...@apache.org> wrote: > Hi Everyone, > > I'd like to discuss about releasing a more full-featured Flink > distribution. The motivation is that there is friction for SQL/Table API > users that want to use Table connectors which are not there in the > current Flink Distribution. For these users the workflow is currently > roughly: > > - download Flink dist > - configure csv/Kafka/json connectors per configuration > - run SQL client or program > - decrypt error message and research the solution > - download additional connector jars > - program works correctly > > I realize that this can be made to work but if every SQL user has this > as their first experience that doesn't seem good to me. > > My proposal is to provide two versions of the Flink Distribution in the > future: "fat" and "slim" (names to be discussed): > > - slim would be even trimmer than todays distribution > - fat would contain a lot of convenience connectors (yet to be > determined which one) > > And yes, I realize that there are already more dimensions of Flink > releases (Scala version and Java version). > > For background, our current Flink dist has these in the opt directory: > > - flink-azure-fs-hadoop-1.10.0.jar > - flink-cep-scala_2.12-1.10.0.jar > - flink-cep_2.12-1.10.0.jar > - flink-gelly-scala_2.12-1.10.0.jar > - flink-gelly_2.12-1.10.0.jar > - flink-metrics-datadog-1.10.0.jar > - flink-metrics-graphite-1.10.0.jar > - flink-metrics-influxdb-1.10.0.jar > - flink-metrics-prometheus-1.10.0.jar > - flink-metrics-slf4j-1.10.0.jar > - flink-metrics-statsd-1.10.0.jar > - flink-oss-fs-hadoop-1.10.0.jar > - flink-python_2.12-1.10.0.jar > - flink-queryable-state-runtime_2.12-1.10.0.jar > - flink-s3-fs-hadoop-1.10.0.jar > - flink-s3-fs-presto-1.10.0.jar > - flink-shaded-netty-tcnative-dynamic-2.0.25.Final-9.0.jar > - flink-sql-client_2.12-1.10.0.jar > - flink-state-processor-api_2.12-1.10.0.jar > - flink-swift-fs-hadoop-1.10.0.jar > > Current Flink dist is 267M. If we removed everything from opt we would > go down to 126M. I would reccomend this, because the large majority of > the files in opt are probably unused. > > What do you think? > > Best, > Aljoscha > >
