Hi Tianqi, dmlc looks really cool and it would be great to integrate it with Flink. As far as I understood your requirements, I think that you can already implement most of it on Flink.
For example, starting a special container which does not receive any input could be a specialized SourceOperator. In this SourceOperator you could then start a parameter server which receives the partial updates of the processing tasks. You're right that each task is executed by its own thread. However, you always can spawn new threads from within your user function. This should allow you to run the ML code multi-threaded. But then it might be advisable to lower the number of slots a bit so that you have more cores available than slots defined. The integration of the dmlc API with the existing ML pipelines should not be too hard. As long as one has access to the resulting data set it should be easy to plug it into another predictor/estimator instance. I guess we would mainly need some tooling around it. Looking forward running xgboost and mxnet with Flink :-) Cheers, Till On Sat, Mar 12, 2016 at 7:17 PM, Simone Robutti < simone.robu...@radicalbit.io> wrote: > Thanks for the insight, what you're doing is really interesting. I will > definitely spend some time looking at DMLC and MXNet. > > 2016-03-12 18:35 GMT+01:00 Tianqi Chen <tqc...@cs.washington.edu>: > > > Thanks for the reply. I am writing a long email to give the answers to > > Simone and clarifies what we do > > > > I want to mention that *you can use the library already in Flink*. See > > Flink example here: > > https://github.com/dmlc/xgboost/tree/master/jvm-packages#xgboost-flink > > > > I have not run pressure test on top of Flink, but we did the pressure > test > > thing on Spark and it is consistent with our standalone version on a 100M > > example dataset, gives around 10x over mllib's version. I assume same > thing > > holds for Flink as well. > > So if you are interested, please try it out. I imagine this can be very > > useful to have a Flink demo that can directly give competitive result on > > say a kaggle competition, and can attract more users to Flink community > as > > well. > > > > *The Internal Details* > > Here at dmlc.ml , we are building libraries that we dive deep and aim > > for the best performance and flexibility. We build our own abstractions > > when needed, for example, XGBoost relies on Allreduce, and MXNet, another > > well known deep learning project, relies on parameter server abstraction. > > We tried to make these abstractions portable, so they are not stand-alone > > C++ programs, but can be used as library in other languages, e.g. scala. > > So essentially, here is what is needed: > > - Start a tracker on driver side to connect the workers together to use > our > > version of communication library (this can be swapped depending on level > of > > integration, if communication API is provided natively by the platform). > > - An API to start concurrent jobs(containers), that can execute a > function > > (either worker or server). > > - Gettng accessed to partition of data in each worker. > > > > Take XGBoost for example, what we do in Flink is to as a MapPartition > > stage, and treat each slot as an worker. Each worker then > collaboratively > > solve the machine learning problem, and return the trained model. > > > > *What is needed from DataFlow * > > Dataflow is a nice abstraction for data processing. As you can see, the > > approach we take is somewhat more low level, I would call it developer > > API. Since the requirement is basically to start the worker other types > of > > containers that runs a scala function from driver side. MapPartition > works > > well for the XGBoost case, but here are what can be improved: > > - Being able to specify resources of the slot at the ML stage, for > example, > > xgboost as well as deep learning program can benefit from using multiple > > cores in each worker. While currently mapPartition uses one core for each > > Parititon. > > - Being able to launch container that does not take data, for example > > parameter server instance. This is mainly needed for the deep learning > > program. > > > > > > *Why not implement them using DataFlow?* > > One thing I can expect people to argue is why not directly use (multiple) > > data-flow stages to implement these algorithm. This is a possible > approach, > > here are the reasons why > > - Most work in advanced ML algorithm is actually the machine learning > part, > > and add a bit communication into it. So directly using communication > > library inside ML code allows easier migration from optimized single > > machine version to distributed one. > > - Not all dataflow executors are alike, for machine learning usually > > benefit from persistent program state (which Flink have but not spark), > and > > we want to be invariant of such difference. > > Dataflow was originally designed for data process, and I do feel > sometimes > > other abstraction fits machine learning well. The idea of embedding the > > ML's abstraction into one stage of dataflow allow us to take benefit from > > the flexible data processing phase, and also use the best learning > > algorithm. > > > > *Fault Tolerance?* > > Most algorithm we have assumes a fail-restart scheme from the host > > platform, which means we will rely on system to restart the failed jobs > > somewhere, and provide the same input data. Then internally the > > communication library will kick in and try to recover, usually via some > > checkpoint. Of course if there is checkpoint feature from the host, this > > can also be used. > > > > > > > > *More Machine Learning Algorithms?* > > XGBoost is part of DMLC http://dmlc.ml project. Our goal is *not to > > *develop > > general library that covers all algorithms, like FlinkML. Instead, we > pick > > all the most important ones which are used in production pipeline, and > > build *deeply optimized for each specific one as a package.* Of course > > there are also shared components like communication library and > duplicated > > effort among the libraries are shared. I believe we covered most things > > people need, plus some simple ones that can be directly implemented in > > FlinkML(Kmeans, linear model). > > > > One thing that could be interesting to try next is MXNet > > https://github.com/dmlc/mxnet/tree/master/scala-package, which is a full > > fledge deep learning library that comes with all the features you need as > > well as a Scala Binding. However, > > we do need a bit more things that I mentioned in the requirement section. > > > > > > *What Help we can get from Flink Community * > > I will list the points that are clear and actionable here: > > > > *- *Improve xgboost-Flink API so that it is consistent with current > FlinkML > > pipeline > > - Provide some "developer API" that allows perf improvement as I > mentioned > > in "What is needed from DataFlow" > > - Support abstraction needed for MXNet, and enable *streaming, > GPU-enabled > > distributed deep learning on Flink* > > - Main obstacle will be the "developer API" > > > > While some of these effort seems to be a lot to port specific machine > > learning library. Enable them basically enable port all machine learning > > libraries we build and we will be building using these abstractions. > > > > Tianqi > > > > > > On Sat, Mar 12, 2016 at 4:51 AM, Theodore Vasiloudis < > > theodoros.vasilou...@gmail.com> wrote: > > > > > Hello Tianqui, > > > > > > Yes that definitely sounds interesting for us and we are looking > forward > > to > > > help out with the implementation. > > > > > > Regards, > > > Theodore > > > -- > > > Sent from a mobile device. May contain autocorrect errors. > > > On Mar 12, 2016 11:29 AM, "Simone Robutti" < > simone.robu...@radicalbit.io > > > > > > wrote: > > > > > > > This is a really interesting approach. The idea of a ML library over > > > > DataFlow is probably a winning move and I hope it will stop the > > > > proliferation of worthless reimplementation that is taking place in > the > > > big > > > > data world. Do you think that DataFlow posed specific problems to > your > > > > work? Does it missing something that you had to fill in with your > work? > > > > > > > > Here at RadicalBit we are interested both in DataFlow/Apache Beam and > > in > > > > distributed ML and your approach to us look the best and I hope more > > and > > > > more teams follow your example, maybe integrating existing libraries > > like > > > > H2O with DataFlow. > > > > > > > > Keep us updated if you plan to develop other algorithms. > > > > > > > > 2016-03-11 21:32 GMT+01:00 Tianqi Chen <tqc...@cs.washington.edu>: > > > > > > > > > Hi Flink Developers > > > > > I am sending this email to let you know about XGBoost4J, a > > package > > > > that > > > > > we are planning to announce next week . Here is the draft version > of > > > the > > > > > post > > > > > > > https://github.com/dmlc/xgboost/blob/master/doc/jvm/xgboost4j-intro.md > > > > > > > > > > In short, XGBoost is a machine learning package that is used by > > > more > > > > > than half of the machine challenge winning solutions and is already > > > > widely > > > > > used in industry. The distributed version scale to billion > > examples(10x > > > > > faster than spark.mllib in the experiment) with fewer resources > (see > > . > > > > > http://arxiv.org/abs/1603.02754) > > > > > > > > > > We are interested in putting distributed XGBoost into all > > Dataflow > > > > > platforms include Flink. This does not mean we re-implement it on > > > Flink. > > > > > But instead we build a portable API that has a communication > library, > > > and > > > > > being able to run on different DataFlow programs. > > > > > > > > > > We hope this can benefit the Flink users, to enable them to get > > > > access > > > > > to one of the state-of-art machine learning algorithm. I am sending > > > this > > > > > email to the mail-list to let you know about it, and hoping to get > > some > > > > > contributors to help improving the XGBoost Flink API to be more > > > > compatible > > > > > with current FlinkML stack. We also hope to get some support from > > the > > > > > system side, to enable some abstraction needed in XGBoost for using > > > > > multiple threads within even one slot for maximum performance. > > > > > > > > > > > > > > > Let us know about your thoughts. > > > > > > > > > > Cheers > > > > > > > > > > Tianqi > > > > > > > > > > > > > > >
