Hi Rong, Thanks a lot for joining the discussion!
It would be great if we can have a long term plan. My intention is to provide a way for users to add dependencies of Flink ML, either through the opt or download page. This would be more and more critical along with the improvement of the Flink ML, as you said there are multiple PRs under review and I'm also going to support Python Pipeline API recently[1]. Meanwhile, it also makes sense to include the API into the opt, so it would probably not break the long term plan. However, even find something wrong in the future, we can revisit this easily instead of blocking the improvement for users. What do you think? Best, Hequn [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Support-Python-ML-Pipeline-API-td37291.html On Sat, Feb 8, 2020 at 1:57 AM Rong Rong <walter...@gmail.com> wrote: > CC @Xu Yang <xuyang1...@gmail.com> > > Thanks for starting the discussion @Hequn Cheng <chenghe...@gmail.com> and > sorry for joining the discussion late. > > I've mainly helped merging the code in flink-ml-api and flink-ml-lib in > the past several months. > IMO the flink-ml-api are an extension on top of the table API and agree > that it should be treated as a part of the "core" core. > > However, I think given the fact that there are multiple PRs still under > review [1], is it a better idea to come up with a long term plan first > before make the decision to moving it to /opt now? > > > -- > Rong > > [1] > https://github.com/apache/flink/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aopen+label%3Acomponent%3DLibrary%2FMachineLearning+ > > On Fri, Feb 7, 2020 at 5:54 AM Hequn Cheng <he...@apache.org> wrote: > >> Hi, >> >> @Till Rohrmann <trohrm...@apache.org> Thanks for the great inputs. I >> agree >> with you that we should have a long term plan for this. It definitely >> deserves another discussion. >> @Jeff Zhang <zjf...@gmail.com> Thanks for your reports and ideas. It's a >> good idea to improve the error messages. Do we have any JIRAs for it or >> maybe we can create one for it. >> >> Thank you again for your feedback and suggestions. I will go on with the >> PR. Thanks! >> >> Best, >> Hequn >> >> On Thu, Feb 6, 2020 at 11:51 PM Jeff Zhang <zjf...@gmail.com> wrote: >> >> > I have another concern which may not be closely related to this thread. >> > Since flink doesn't include all the necessary jars, I think it is >> critical >> > for flink to display meaningful error message when any class is missing. >> > e.g. Here's the error message when I use kafka but miss >> > including flink-json. To be honest, the kind of error message is hard >> to >> > understand for new users. >> > >> > >> > Reason: No factory implements >> > 'org.apache.flink.table.factories.DeserializationSchemaFactory'. The >> > following properties are requested: >> > connector.properties.bootstrap.servers=localhost:9092 >> > connector.properties.group.id=testGroup >> > connector.properties.zookeeper.connect=localhost:2181 >> > connector.startup-mode=earliest-offset connector.topic=generated.events >> > connector.type=kafka connector.version=universal format.type=json >> > schema.0.data-type=VARCHAR(2147483647) schema.0.name=status >> > schema.1.data-type=VARCHAR(2147483647) schema.1.name=direction >> > schema.2.data-type=BIGINT schema.2.name=event_ts update-mode=append The >> > following factories have been considered: >> > org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory >> > org.apache.flink.table.module.hive.HiveModuleFactory >> > org.apache.flink.table.module.CoreModuleFactory >> > org.apache.flink.table.catalog.GenericInMemoryCatalogFactory >> > org.apache.flink.table.sources.CsvBatchTableSourceFactory >> > org.apache.flink.table.sources.CsvAppendTableSourceFactory >> > org.apache.flink.table.sinks.CsvBatchTableSinkFactory >> > org.apache.flink.table.sinks.CsvAppendTableSinkFactory >> > org.apache.flink.table.planner.delegation.BlinkPlannerFactory >> > org.apache.flink.table.planner.delegation.BlinkExecutorFactory >> > org.apache.flink.table.planner.StreamPlannerFactory >> > org.apache.flink.table.executor.StreamExecutorFactory >> > org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactory >> at >> > >> > >> org.apache.flink.table.factories.TableFactoryService.filterByFactoryClass(TableFactoryService.java:238) >> > at >> > >> > >> org.apache.flink.table.factories.TableFactoryService.filter(TableFactoryService.java:185) >> > at >> > >> > >> org.apache.flink.table.factories.TableFactoryService.findSingleInternal(TableFactoryService.java:143) >> > at >> > >> > >> org.apache.flink.table.factories.TableFactoryService.find(TableFactoryService.java:113) >> > at >> > >> > >> org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.getDeserializationSchema(KafkaTableSourceSinkFactoryBase.java:277) >> > at >> > >> > >> org.apache.flink.streaming.connectors.kafka.KafkaTableSourceSinkFactoryBase.createStreamTableSource(KafkaTableSourceSinkFactoryBase.java:161) >> > at >> > >> > >> org.apache.flink.table.factories.StreamTableSourceFactory.createTableSource(StreamTableSourceFactory.java:49) >> > at >> > >> > >> org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSource(TableFactoryUtil.java:53) >> > ... 36 more >> > >> > >> > >> > Till Rohrmann <trohrm...@apache.org> 于2020年2月6日周四 下午11:30写道： >> > >> > > I would not object given that it is rather small at the moment. >> However, >> > I >> > > also think that we should have a plan how to handle the ever growing >> > Flink >> > > ecosystem and how to make it easily accessible to our users. E.g. one >> far >> > > fetched idea could be something like a configuration script which >> > downloads >> > > the required components for the user. But this deserves definitely a >> > > separate discussion and does not really belong here. >> > > >> > > Cheers, >> > > Till >> > > >> > > On Thu, Feb 6, 2020 at 3:35 PM Hequn Cheng <he...@apache.org> wrote: >> > > >> > > > >> > > > Hi everyone, >> > > > >> > > > Thank you all for the great inputs! >> > > > >> > > > I think probably what we all agree on is we should try to make a >> leaner >> > > > flink-dist. However, we may also need to do some compromises >> > considering >> > > > the user experience that users don't need to download the >> dependencies >> > > from >> > > > different places. Otherwise, we can move all the jars in the current >> > opt >> > > > folder to the download page. >> > > > >> > > > The missing of clear rules for guiding such compromises makes things >> > more >> > > > complicated now. I would agree that the decisive factor for what >> goes >> > > into >> > > > Flink's binary distribution should be how core it is to Flink. >> > Meanwhile, >> > > > it's better to treat Flink API as a (core) core to Flink. Not only >> it >> > is >> > > a >> > > > very clear rule that easy to be followed but also in most cases, >> API is >> > > > very significant and deserved to be included in the dist. >> > > > >> > > > Given this, it might make sense to put flink-ml-api and flink-ml-lib >> > into >> > > > the opt. >> > > > What do you think? >> > > > >> > > > Best, >> > > > Hequn >> > > > >> > > > On Wed, Feb 5, 2020 at 12:39 AM Chesnay Schepler < >> ches...@apache.org> >> > > > wrote: >> > > > >> > > >> Around a year ago I started a discussion >> > > >> < >> > > >> > >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Towards-a-leaner-flink-dist-tp25615.html >> > > > >> > > >> on reducing the amount of jars we ship with the distribution. >> > > >> >> > > >> While there was no definitive conclusion there was a shared >> sentiment >> > > >> that APIs should be shipped with the distribution. >> > > >> >> > > >> On 04/02/2020 17:25, Till Rohrmann wrote: >> > > >> >> > > >> I think there is no such rule that APIs go automatically into opt/ >> and >> > > >> "libraries" not. The contents of opt/ have mainly grown over time >> w/o >> > > >> following a strict rule. >> > > >> >> > > >> I think the decisive factor for what goes into Flink's binary >> > > distribution >> > > >> should be how core it is to Flink. Of course another important >> > > >> consideration is which use cases Flink should promote "out of the >> box" >> > > (not >> > > >> sure whether this is actual true for content shipped in opt/ >> because >> > you >> > > >> also have to move it to lib). >> > > >> >> > > >> For example, Gelly would be an example which I would rather see as >> an >> > > >> optional component than shipping it with every Flink binary >> > > distribution. >> > > >> >> > > >> Cheers, >> > > >> Till >> > > >> >> > > >> On Tue, Feb 4, 2020 at 11:24 AM Becket Qin <becket....@gmail.com> >> < >> > > becket....@gmail.com> wrote: >> > > >> >> > > >> >> > > >> Thanks for the suggestion, Till. >> > > >> >> > > >> I am curious about how do we usually decide when to put the jars >> into >> > > the >> > > >> opt folder? >> > > >> >> > > >> Technically speaking, it seems that `flink-ml-api` should be put >> into >> > > the >> > > >> opt directory because they are actually API instead of libraries, >> just >> > > like >> > > >> CEP and Table. >> > > >> >> > > >> `flink-ml-lib` seems to be on the border. On one hand, it is a >> > library. >> > > On >> > > >> the other hand, unlike SQL formats and Hadoop whose major code are >> > > outside >> > > >> of Flink, the algorithm codes are in Flink. So `flink-ml-lib` is >> more >> > > like >> > > >> those of built-in SQL UDFs. So it seems fine to either put it in >> the >> > opt >> > > >> folder or in the downloads page. >> > > >> >> > > >> From the user experience perspective, it might be better to have >> both >> > > >> `flink-ml-lib` and `flink-ml-api` in opt folder so users needn't >> go to >> > > two >> > > >> places for the required dependencies. >> > > >> >> > > >> Thanks, >> > > >> >> > > >> Jiangjie (Becket) Qin >> > > >> >> > > >> On Tue, Feb 4, 2020 at 2:32 PM Hequn Cheng <he...@apache.org> < >> > > he...@apache.org> wrote: >> > > >> >> > > >> >> > > >> Hi Till, >> > > >> >> > > >> Thanks a lot for your suggestion. It's a good idea to offer the >> > flink-ml >> > > >> libraries as optional dependencies on the download page which can >> make >> > > >> >> > > >> the >> > > >> >> > > >> dist smaller. >> > > >> >> > > >> But I also have some concerns for it, e.g., the download page now >> only >> > > >> includes the latest 3 releases. We may need to find ways to support >> > more >> > > >> versions. >> > > >> On the other hand, the size of the flink-ml libraries now is very >> > > >> small(about 246K), so it would not bring much impact on the size of >> > > dist. >> > > >> >> > > >> What do you think? >> > > >> >> > > >> Best, >> > > >> Hequn >> > > >> >> > > >> On Mon, Feb 3, 2020 at 6:24 PM Till Rohrmann <trohrm...@apache.org> >> < >> > > trohrm...@apache.org> >> > > >> >> > > >> wrote: >> > > >> >> > > >> An alternative solution would be to offer the flink-ml libraries as >> > > >> optional dependencies on the download page. Similar to how we offer >> > the >> > > >> different SQL formats and Hadoop releases [1]. >> > > >> >> > > >> [1] https://flink.apache.org/downloads.html >> > > >> >> > > >> Cheers, >> > > >> Till >> > > >> >> > > >> On Mon, Feb 3, 2020 at 10:19 AM Hequn Cheng <he...@apache.org> < >> > > he...@apache.org> wrote: >> > > >> >> > > >> >> > > >> Thank you all for your feedback and suggestions! >> > > >> >> > > >> Best, Hequn >> > > >> >> > > >> On Mon, Feb 3, 2020 at 5:07 PM Becket Qin <becket....@gmail.com> < >> > > becket....@gmail.com> >> > > >> >> > > >> wrote: >> > > >> >> > > >> Thanks for bringing up the discussion, Hequn. >> > > >> >> > > >> +1 on adding `flink-ml-api` and `flink-ml-lib` into opt. This would >> > > >> >> > > >> make >> > > >> >> > > >> it much easier for the users to try out some simple ml tasks. >> > > >> >> > > >> Thanks, >> > > >> >> > > >> Jiangjie (Becket) Qin >> > > >> >> > > >> On Mon, Feb 3, 2020 at 4:34 PM jincheng sun < >> > > >> >> > > >> sunjincheng...@gmail.com >> > > >> >> > > >> wrote: >> > > >> >> > > >> >> > > >> Thank you for pushing forward @Hequn Cheng <he...@apache.org> < >> > > he...@apache.org> ! >> > > >> >> > > >> Hi @Becket Qin <becket....@gmail.com> <becket....@gmail.com> , Do >> > you >> > > have any concerns >> > > >> >> > > >> on >> > > >> >> > > >> this ? >> > > >> >> > > >> Best, >> > > >> Jincheng >> > > >> >> > > >> Hequn Cheng <he...@apache.org> <he...@apache.org> 于2020年2月3日周一 >> > > 下午2:09写道： >> > > >> >> > > >> >> > > >> Hi everyone, >> > > >> >> > > >> Thanks for the feedback. As there are no objections, I've opened a >> > > >> >> > > >> JIRA >> > > >> >> > > >> issue(FLINK-15847[1]) to address this issue. >> > > >> The implementation details can be discussed in the issue or in the >> > > >> following PR. >> > > >> >> > > >> Best, >> > > >> Hequn >> > > >> >> > > >> [1] https://issues.apache.org/jira/browse/FLINK-15847 >> > > >> >> > > >> On Wed, Jan 8, 2020 at 9:15 PM Hequn Cheng <chenghe...@gmail.com> >> < >> > > chenghe...@gmail.com> >> > > >> >> > > >> wrote: >> > > >> >> > > >> Hi Jincheng, >> > > >> >> > > >> Thanks a lot for your feedback! >> > > >> Yes, I agree with you. There are cases that multi jars need to >> > > >> >> > > >> be >> > > >> >> > > >> uploaded. I will prepare another discussion later. Maybe with a >> > > >> >> > > >> simple >> > > >> >> > > >> design doc. >> > > >> >> > > >> Best, Hequn >> > > >> >> > > >> On Wed, Jan 8, 2020 at 3:06 PM jincheng sun < >> > > >> >> > > >> sunjincheng...@gmail.com> >> > > >> >> > > >> wrote: >> > > >> >> > > >> >> > > >> Thanks for bring up this discussion Hequn! >> > > >> >> > > >> +1 for include `flink-ml-api` and `flink-ml-lib` in opt. >> > > >> >> > > >> BTW: I think would be great if bring up a discussion for upload >> > > >> >> > > >> multiple >> > > >> >> > > >> Jars at the same time. as PyFlink JOB also can have the benefit >> > > >> >> > > >> if >> > > >> >> > > >> we >> > > >> >> > > >> do >> > > >> >> > > >> that improvement. >> > > >> >> > > >> Best, >> > > >> Jincheng >> > > >> >> > > >> >> > > >> Hequn Cheng <chenghe...@gmail.com> <chenghe...@gmail.com> >> > 于2020年1月8日周三 >> > > 上午11:50写道： >> > > >> >> > > >> >> > > >> Hi everyone, >> > > >> >> > > >> FLIP-39[1] rebuilds Flink ML pipeline on top of TableAPI >> > > >> >> > > >> which >> > > >> >> > > >> moves >> > > >> >> > > >> Flink >> > > >> >> > > >> ML a step further. Base on it, users can develop their ML >> > > >> >> > > >> jobs >> > > >> >> > > >> and >> > > >> >> > > >> more >> > > >> >> > > >> and >> > > >> >> > > >> more machine learning platforms are providing ML services. >> > > >> >> > > >> However, the problem now is the jars of flink-ml-api and >> > > >> >> > > >> flink-ml-lib >> > > >> >> > > >> are >> > > >> >> > > >> only exist on maven repo. Whenever users want to submit ML >> > > >> >> > > >> jobs, >> > > >> >> > > >> they >> > > >> >> > > >> can >> > > >> >> > > >> only depend on the ml modules and package a fat jar. This >> > > >> >> > > >> would be >> > > >> >> > > >> inconvenient especially for the machine learning platforms on >> > > >> >> > > >> which >> > > >> >> > > >> nearly >> > > >> >> > > >> all jobs depend on Flink ML modules and have to package a fat >> > > >> >> > > >> jar. >> > > >> >> > > >> Given this, it would be better to include jars of >> > > >> >> > > >> flink-ml-api >> > > >> >> > > >> and >> > > >> >> > > >> flink-ml-lib in the `opt` folder, so that users can directly >> > > >> >> > > >> use >> > > >> >> > > >> the >> > > >> >> > > >> jars >> > > >> >> > > >> with the binary release. For example, users can move the jars >> > > >> >> > > >> into >> > > >> >> > > >> the >> > > >> >> > > >> `lib` folder or use -j to upload the jars. (Currently, -j >> > > >> >> > > >> only >> > > >> >> > > >> support >> > > >> >> > > >> upload one jar. Supporting multi jars for -j can be discussed >> > > >> >> > > >> in >> > > >> >> > > >> another >> > > >> >> > > >> discussion.) >> > > >> >> > > >> Putting the jars in the `opt` folder instead of the `lib` >> > > >> >> > > >> folder >> > > >> >> > > >> is >> > > >> >> > > >> because >> > > >> >> > > >> currently, the ml jars are still optional for the Flink >> > > >> >> > > >> project by >> > > >> >> > > >> default. >> > > >> >> > > >> What do you think? Welcome any feedback! >> > > >> >> > > >> Best, >> > > >> >> > > >> Hequn >> > > >> >> > > >> [1] >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> > >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs >> > > >> >> > > >> >> > > >> >> > > >> > >> > >> > -- >> > Best Regards >> > >> > Jeff Zhang >> > >> >
