Ok, agree, that I should start discussion before making changes, but I was limited by release 2.8 and trying don;'t be a delayed person for that. During release I was focused on fixing bugs and don't tests TF and Ignite together
I thought that as a maintainer of ML module I could do perform these actions. Below I will share my statement why it should be removed and why it should be removed immediately About TensorFlow module (reason for removal) 1. This module is only one module that uses IGFS and needs in FileSystem on Ignte side due to TensorFlow API 2. This module a part of bridge between Ignite ML and Tensorflow and its broken after changes in TensorFlow on TensorFlow side 3. TensorFlow released new version without Ignite bridge, no chance to run them together for new releases 4. This module wasn't complete and developer who did this, left the community 5. The development skills for this story require python/C++/java programming together 6. The module is a source of bugs which could be fixed for release 2.8 and possibly for future releases (nobody in community could this) 7. The release size reduced from 6 Gb to 4.5 due to removed dependencies 8. TensorFlow now is not popular among Data Scientists, the PyTorch is the most popular tool for Deep Learning (like NetBeans and IDEA) 9. Nobody uses that in production because it was developed between 2.7 and 2.8 (2.7 has only proof-of-concept) Nikolay, sorry for that, hope to share more information about the ML and discuss here the main changes before actions. пн, 17 февр. 2020 г. в 16:18, Nikolay Izhikov <nizhi...@apache.org>: > Hello, Alexey. > > > The main reason, the modules are not work proper way, were experimental, > > never released as a production-ready, support old, outdated version, the > > external frameworks, like Tensorflow, move integration with ignite to the > > special repos, they are not finished, the code there is broken and > couldn't > > be fixed, because and I have no power/C++ skills/permission to commit > > something to them and time to support this broken modules. > > > Do we have some tickets or wider explanation for it? > It very uncommon for me that the decision to remove modules from the > master and release is not discussed widely in the community. > > > 17 февр. 2020 г., в 14:39, Ravil Galeyev <rgale...@gmail.com> > написал(а): > > > > Hi Team, > > > > First of all, let me introduce myself. I’m Ravil, I contribute to the ML > > module since 2018 and from time to time I make talks about it. (I..e data > > science summit in Warsaw [1]). > > > > So, Alexey made a huge effort to develop the ML module but he is not > alone. > > If you check the repo you will find other contributors. > > > > Therefore the ML module is alive and is able to run and has the roadmap. > > For me, it means that it’s not a raw project. > > > > Regarding documentation, it’d like to mention the code is the best > > documentation :) > > > > We have examples for most algorithms [2]. But if it needed I’m ready to > > help the community with documentation in English German Polish or > Russain. > > > > > > [1] https://dssconf.pl/ > > > > [2] > > > https://github.com/apache/ignite/tree/master/examples/src/main/java/org/apache/ignite/examples/ml > > > > Best regards, > > > > Ravil > > > > > > On Mon, 17 Feb 2020 at 11:49, Alexey Zinoviev <zaleslaw....@gmail.com> > > wrote: > > > >> Hello, Igniters, and you, Nikolay. > >> > >> First of all, if you have real interest to the ML module and its state, > I > >> could make call with you and explain this. > >> > >> > >> *As far as I know, for now, we have only 1 active contributor to this > area > >> -Alexey Zinoviev.* > >> Currently, we have 2 active contributors, me and Ravil Galeeyev, a few > >> newbies, another guys who started tensorflow and another modules and > >> submodules don't visit the community for many months. > >> > >> *Is ML module production ready?* > >> This release will be the first release, than ML is production ready and > it > >> totally my work. > >> > >> > >> *Can someone related to the ML, please, give some examples of the CVE > >> orissues that can be fixed only with removing a bunch of modules?* > >> CVE is not the main reason to remove the "bunch of modules", but part of > >> the story. > >> The main reason, the modules are not work proper way, were experimental, > >> never released as a production-ready, support old, outdated version, the > >> external frameworks, like Tensorflow, move integration with ignite to > the > >> special repos, they are not finished, the code there is broken and > couldn't > >> be fixed, because and I have no power/C++ skills/permission to commit > >> something to them and time to support this broken modules. > >> > >> Also broken TF module blocks the removal of IGFS. > >> > >> Found CVE were related to the dependencies related to hadoop/tf/parquet > and > >> so on. > >> > >> *Should we mark it with the @IgniteExperimental? * > >> I don't know, we have no this RAW annotation a few weeks ago and I don't > >> know how we should use it. > >> It could be, if you finish the discussion about this annotation and the > >> write docs about it and share it with me. > >> > >> > >> * As far as I know, the ML module has no documentation. Is it correct? > Dowe > >> have plans to fix it?* > >> The ML docs are here, on our Ignite documentation > >> https://apacheignite.readme.io/docs/machine-learning > >> of course, something could be wrong, 1.5 year we are not released Ignite > >> Yes, I have plan to fix. Of course, after fixing all bugs in release > branch > >> > >> *Should we move it to the ignite-extensions?* > >> No we shouldn't, I don't want this, and have a lot of arguments and > >> currently is not the time for discussion about this (they are too young > and > >> have now real infrastructure and release-cycle) > >> > >> P.S. Community, I understand that the removal of module looks strange, > but > >> we could understand that ML was a strange experiment without roadmap and > >> this situation is finished. > >> Now, I have roadmap (will be published later), newbie tickets, the > ability > >> to prepare correct docs, the understanding what could be used by clients > >> and first of all, production-ready ML (it could be run on > Ignite-cluster, > >> really, it works.) > >> > >> If you, Igniters, believe, that I could be a good maintainer for ML > module, > >> please support me here in this thread > >> If you think, that I do something wrong, OK, please write it too, I'll > read > >> carefully. > >> > >> I spent a few months to fix bugs in components, which were abandoned by > >> their creators. > >> > >> My goal: Ignite should have light-weight, easy-integrated ML without > >> strange and unfinished experiments which could be not maintained. It's a > >> part of common movement in Ignite (removal of modules or moving them to > the > >> separate repos). > >> > >> > >> пн, 17 февр. 2020 г. в 12:10, Nikolay Izhikov <nizhi...@apache.org>: > >> > >>> Hello, Igniters. > >>> > >>> Can someone bring some light on the state of the ML module in Ignite? > >>> As far as I know, for now, we have only 1 active contributor to this > >> area - > >>> Alexey Zinoviev. > >>> I see how whole modules come and go from the module - [1] > >>> > >>> Please, also note this quote: > >>> > >>>> Also as a result of good testing from both side (from me and Stepan) > we > >>>> found a lot of bugs and CVEs in hadoop related components that should > >> be > >>>> removed in release branch too. > >>> > >>> 0. As far as I know, the ML module has no documentation. Is it correct? > >> Do > >>> we have plans to fix it? > >>> > >>> 1. Can someone related to the ML, please, give some examples of the CVE > >> or > >>> issues that can be fixed only with removing a bunch of modules? > >>> > >>> 3. Is ML module production ready? > >>> > >>> 4. Should we mark it with the @IgniteExperimental? > >>> > >>> 5. Should we move it to the ignite-extensions? > >>> > >>> [1] > >>> > >>> > >> > https://github.com/apache/ignite/commit/53e886b8ed38a6842cef8b44ace6851855dfad29 > >>> > >> > >
