CC @Xu Yang <xuyang1...@gmail.com> Thanks for starting the discussion @Hequn Cheng <chenghe...@gmail.com> and sorry for joining the discussion late.
I've mainly helped merging the code in flink-ml-api and flink-ml-lib in the past several months. IMO the flink-ml-api are an extension on top of the table API and agree that it should be treated as a part of the "core" core. However, I think given the fact that there are multiple PRs still under review [1], is it a better idea to come up with a long term plan first before make the decision to moving it to /opt now? -- Rong [1] https://github.com/apache/flink/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+label%3Acomponent%3DLibrary%2FMachineLearning+ On Fri, Feb 7, 2020 at 5:54 AM Hequn Cheng <he...@apache.org> wrote: > Hi, > > @Till Rohrmann <trohrm...@apache.org> Thanks for the great inputs. I agree > with you that we should have a long term plan for this. It definitely > deserves another discussion. > @Jeff Zhang <zjf...@gmail.com> Thanks for your reports and ideas. It's a > good idea to improve the error messages. Do we have any JIRAs for it or > maybe we can create one for it. > > Thank you again for your feedback and suggestions. I will go on with the > PR. Thanks! > > Best, > Hequn > > On Thu, Feb 6, 2020 at 11:51 PM Jeff Zhang <zjf...@gmail.com> wrote: > > > I have another concern which may not be closely related to this thread. > > Since flink doesn't include all the necessary jars, I think it is > critical > > for flink to display meaningful error message when any class is missing. > > e.g. Here's the error message when I use kafka but miss > > including flink-json. To be honest, the kind of error message is hard to > > understand for new users. > > > > > > Reason: No factory implements > > 'org.apache.flink.table.factories.DeserializationSchemaFactory'. The > > following properties are requested: > > connector.properties.bootstrap.servers=localhost:9092 > > connector.properties.group.id=testGroup > > connector.properties.zookeeper.connect=localhost:2181 > > connector.startup-mode=earliest-offset connector.topic=generated.events > > connector.type=kafka connector.version=universal format.type=json > > schema.0.data-type=VARCHAR(2147483647) schema.0.name=status > > schema.1.data-type=VARCHAR(2147483647) schema.1.name=direction > > schema.2.data-type=BIGINT schema.2.name=event_ts update-mode=append The > > following factories have been considered: > > org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory > > org.apache.flink.table.module.hive.HiveModuleFactory > > org.apache.flink.table.module.CoreModuleFactory > > org.apache.flink.table.catalog.GenericInMemoryCatalogFactory > > org.apache.flink.table.sources.CsvBatchTableSourceFactory > > org.apache.flink.table.sources.CsvAppendTableSourceFactory > > org.apache.flink.table.sinks.CsvBatchTableSinkFactory > > org.apache.flink.table.sinks.CsvAppendTableSinkFactory > > org.apache.flink.table.planner.delegation.BlinkPlannerFactory > > org.apache.flink.table.planner.delegation.BlinkExecutorFactory > > org.apache.flink.table.planner.StreamPlannerFactory > > org.apache.flink.table.executor.StreamExecutorFactory > > org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactory > at > > > > > org.apache.flink.table.factories.TableFactoryService.filterByFactoryClass(TableFactoryService.java:238) > > at > > > > > org.apache.flink.table.factories.TableFactoryService.filter(TableFactoryService.java:185) > > at > > > > > org.apache.flink.table.factories.TableFactoryService.findSingleInternal(TableFactoryService.java:143) > > at > > > > > org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.java:113) > > at > > > > > org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.getDeserializationSchema(KafkaTableSourceSinkFactoryBase.java:277) > > at > > > > > org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.createStreamTableSource(KafkaTableSourceSinkFactoryBase.java:161) > > at > > > > > org.apache.flink.table.factories.StreamTableSourceFactory.createTableSource(StreamTableSourceFactory.java:49) > > at > > > > > org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:53) > > ... 36 more > > > > > > > > Till Rohrmann <trohrm...@apache.org> 于2020年2月6日周四 下午11:30写道： > > > > > I would not object given that it is rather small at the moment. > However, > > I > > > also think that we should have a plan how to handle the ever growing > > Flink > > > ecosystem and how to make it easily accessible to our users. E.g. one > far > > > fetched idea could be something like a configuration script which > > downloads > > > the required components for the user. But this deserves definitely a > > > separate discussion and does not really belong here. > > > > > > Cheers, > > > Till > > > > > > On Thu, Feb 6, 2020 at 3:35 PM Hequn Cheng <he...@apache.org> wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > Thank you all for the great inputs! > > > > > > > > I think probably what we all agree on is we should try to make a > leaner > > > > flink-dist. However, we may also need to do some compromises > > considering > > > > the user experience that users don't need to download the > dependencies > > > from > > > > different places. Otherwise, we can move all the jars in the current > > opt > > > > folder to the download page. > > > > > > > > The missing of clear rules for guiding such compromises makes things > > more > > > > complicated now. I would agree that the decisive factor for what goes > > > into > > > > Flink's binary distribution should be how core it is to Flink. > > Meanwhile, > > > > it's better to treat Flink API as a (core) core to Flink. Not only it > > is > > > a > > > > very clear rule that easy to be followed but also in most cases, API > is > > > > very significant and deserved to be included in the dist. > > > > > > > > Given this, it might make sense to put flink-ml-api and flink-ml-lib > > into > > > > the opt. > > > > What do you think? > > > > > > > > Best, > > > > Hequn > > > > > > > > On Wed, Feb 5, 2020 at 12:39 AM Chesnay Schepler <ches...@apache.org > > > > > > wrote: > > > > > > > >> Around a year ago I started a discussion > > > >> < > > > > > > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Towards-a-leaner-flink-dist-tp25615.html > > > > > > > >> on reducing the amount of jars we ship with the distribution. > > > >> > > > >> While there was no definitive conclusion there was a shared > sentiment > > > >> that APIs should be shipped with the distribution. > > > >> > > > >> On 04/02/2020 17:25, Till Rohrmann wrote: > > > >> > > > >> I think there is no such rule that APIs go automatically into opt/ > and > > > >> "libraries" not. The contents of opt/ have mainly grown over time > w/o > > > >> following a strict rule. > > > >> > > > >> I think the decisive factor for what goes into Flink's binary > > > distribution > > > >> should be how core it is to Flink. Of course another important > > > >> consideration is which use cases Flink should promote "out of the > box" > > > (not > > > >> sure whether this is actual true for content shipped in opt/ because > > you > > > >> also have to move it to lib). > > > >> > > > >> For example, Gelly would be an example which I would rather see as > an > > > >> optional component than shipping it with every Flink binary > > > distribution. > > > >> > > > >> Cheers, > > > >> Till > > > >> > > > >> On Tue, Feb 4, 2020 at 11:24 AM Becket Qin <becket....@gmail.com> < > > > becket....@gmail.com> wrote: > > > >> > > > >> > > > >> Thanks for the suggestion, Till. > > > >> > > > >> I am curious about how do we usually decide when to put the jars > into > > > the > > > >> opt folder? > > > >> > > > >> Technically speaking, it seems that `flink-ml-api` should be put > into > > > the > > > >> opt directory because they are actually API instead of libraries, > just > > > like > > > >> CEP and Table. > > > >> > > > >> `flink-ml-lib` seems to be on the border. On one hand, it is a > > library. > > > On > > > >> the other hand, unlike SQL formats and Hadoop whose major code are > > > outside > > > >> of Flink, the algorithm codes are in Flink. So `flink-ml-lib` is > more > > > like > > > >> those of built-in SQL UDFs. So it seems fine to either put it in the > > opt > > > >> folder or in the downloads page. > > > >> > > > >> From the user experience perspective, it might be better to have > both > > > >> `flink-ml-lib` and `flink-ml-api` in opt folder so users needn't go > to > > > two > > > >> places for the required dependencies. > > > >> > > > >> Thanks, > > > >> > > > >> Jiangjie (Becket) Qin > > > >> > > > >> On Tue, Feb 4, 2020 at 2:32 PM Hequn Cheng <he...@apache.org> < > > > he...@apache.org> wrote: > > > >> > > > >> > > > >> Hi Till, > > > >> > > > >> Thanks a lot for your suggestion. It's a good idea to offer the > > flink-ml > > > >> libraries as optional dependencies on the download page which can > make > > > >> > > > >> the > > > >> > > > >> dist smaller. > > > >> > > > >> But I also have some concerns for it, e.g., the download page now > only > > > >> includes the latest 3 releases. We may need to find ways to support > > more > > > >> versions. > > > >> On the other hand, the size of the flink-ml libraries now is very > > > >> small(about 246K), so it would not bring much impact on the size of > > > dist. > > > >> > > > >> What do you think? > > > >> > > > >> Best, > > > >> Hequn > > > >> > > > >> On Mon, Feb 3, 2020 at 6:24 PM Till Rohrmann <trohrm...@apache.org> > < > > > trohrm...@apache.org> > > > >> > > > >> wrote: > > > >> > > > >> An alternative solution would be to offer the flink-ml libraries as > > > >> optional dependencies on the download page. Similar to how we offer > > the > > > >> different SQL formats and Hadoop releases [1]. > > > >> > > > >> [1] https://flink.apache.org/downloads.html > > > >> > > > >> Cheers, > > > >> Till > > > >> > > > >> On Mon, Feb 3, 2020 at 10:19 AM Hequn Cheng <he...@apache.org> < > > > he...@apache.org> wrote: > > > >> > > > >> > > > >> Thank you all for your feedback and suggestions! > > > >> > > > >> Best, Hequn > > > >> > > > >> On Mon, Feb 3, 2020 at 5:07 PM Becket Qin <becket....@gmail.com> < > > > becket....@gmail.com> > > > >> > > > >> wrote: > > > >> > > > >> Thanks for bringing up the discussion, Hequn. > > > >> > > > >> +1 on adding `flink-ml-api` and `flink-ml-lib` into opt. This would > > > >> > > > >> make > > > >> > > > >> it much easier for the users to try out some simple ml tasks. > > > >> > > > >> Thanks, > > > >> > > > >> Jiangjie (Becket) Qin > > > >> > > > >> On Mon, Feb 3, 2020 at 4:34 PM jincheng sun < > > > >> > > > >> sunjincheng...@gmail.com > > > >> > > > >> wrote: > > > >> > > > >> > > > >> Thank you for pushing forward @Hequn Cheng <he...@apache.org> < > > > he...@apache.org> ! > > > >> > > > >> Hi @Becket Qin <becket....@gmail.com> <becket....@gmail.com> , Do > > you > > > have any concerns > > > >> > > > >> on > > > >> > > > >> this ? > > > >> > > > >> Best, > > > >> Jincheng > > > >> > > > >> Hequn Cheng <he...@apache.org> <he...@apache.org> 于2020年2月3日周一 > > > 下午2:09写道： > > > >> > > > >> > > > >> Hi everyone, > > > >> > > > >> Thanks for the feedback. As there are no objections, I've opened a > > > >> > > > >> JIRA > > > >> > > > >> issue(FLINK-15847[1]) to address this issue. > > > >> The implementation details can be discussed in the issue or in the > > > >> following PR. > > > >> > > > >> Best, > > > >> Hequn > > > >> > > > >> [1] https://issues.apache.org/jira/browse/FLINK-15847 > > > >> > > > >> On Wed, Jan 8, 2020 at 9:15 PM Hequn Cheng <chenghe...@gmail.com> < > > > chenghe...@gmail.com> > > > >> > > > >> wrote: > > > >> > > > >> Hi Jincheng, > > > >> > > > >> Thanks a lot for your feedback! > > > >> Yes, I agree with you. There are cases that multi jars need to > > > >> > > > >> be > > > >> > > > >> uploaded. I will prepare another discussion later. Maybe with a > > > >> > > > >> simple > > > >> > > > >> design doc. > > > >> > > > >> Best, Hequn > > > >> > > > >> On Wed, Jan 8, 2020 at 3:06 PM jincheng sun < > > > >> > > > >> sunjincheng...@gmail.com> > > > >> > > > >> wrote: > > > >> > > > >> > > > >> Thanks for bring up this discussion Hequn! > > > >> > > > >> +1 for include `flink-ml-api` and `flink-ml-lib` in opt. > > > >> > > > >> BTW: I think would be great if bring up a discussion for upload > > > >> > > > >> multiple > > > >> > > > >> Jars at the same time. as PyFlink JOB also can have the benefit > > > >> > > > >> if > > > >> > > > >> we > > > >> > > > >> do > > > >> > > > >> that improvement. > > > >> > > > >> Best, > > > >> Jincheng > > > >> > > > >> > > > >> Hequn Cheng <chenghe...@gmail.com> <chenghe...@gmail.com> > > 于2020年1月8日周三 > > > 上午11:50写道： > > > >> > > > >> > > > >> Hi everyone, > > > >> > > > >> FLIP-39[1] rebuilds Flink ML pipeline on top of TableAPI > > > >> > > > >> which > > > >> > > > >> moves > > > >> > > > >> Flink > > > >> > > > >> ML a step further. Base on it, users can develop their ML > > > >> > > > >> jobs > > > >> > > > >> and > > > >> > > > >> more > > > >> > > > >> and > > > >> > > > >> more machine learning platforms are providing ML services. > > > >> > > > >> However, the problem now is the jars of flink-ml-api and > > > >> > > > >> flink-ml-lib > > > >> > > > >> are > > > >> > > > >> only exist on maven repo. Whenever users want to submit ML > > > >> > > > >> jobs, > > > >> > > > >> they > > > >> > > > >> can > > > >> > > > >> only depend on the ml modules and package a fat jar. This > > > >> > > > >> would be > > > >> > > > >> inconvenient especially for the machine learning platforms on > > > >> > > > >> which > > > >> > > > >> nearly > > > >> > > > >> all jobs depend on Flink ML modules and have to package a fat > > > >> > > > >> jar. > > > >> > > > >> Given this, it would be better to include jars of > > > >> > > > >> flink-ml-api > > > >> > > > >> and > > > >> > > > >> flink-ml-lib in the `opt` folder, so that users can directly > > > >> > > > >> use > > > >> > > > >> the > > > >> > > > >> jars > > > >> > > > >> with the binary release. For example, users can move the jars > > > >> > > > >> into > > > >> > > > >> the > > > >> > > > >> `lib` folder or use -j to upload the jars. (Currently, -j > > > >> > > > >> only > > > >> > > > >> support > > > >> > > > >> upload one jar. Supporting multi jars for -j can be discussed > > > >> > > > >> in > > > >> > > > >> another > > > >> > > > >> discussion.) > > > >> > > > >> Putting the jars in the `opt` folder instead of the `lib` > > > >> > > > >> folder > > > >> > > > >> is > > > >> > > > >> because > > > >> > > > >> currently, the ml jars are still optional for the Flink > > > >> > > > >> project by > > > >> > > > >> default. > > > >> > > > >> What do you think? Welcome any feedback! > > > >> > > > >> Best, > > > >> > > > >> Hequn > > > >> > > > >> [1] > > > >> > > > >> > > > >> > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs > > > >> > > > >> > > > >> > > > > > > > > > -- > > Best Regards > > > > Jeff Zhang > > >
