Continuation on last email sent by mistake: Is cpl license is compatible with apache ?
http://opensource.org/licenses/cpl1.0.php Mallet jars are available on maven. They have hessian based solvers which looked interesting along with bfgs and cg. Definitely the lbfgs f2j looks promising as the base fortran code is perhaps the best bfgs code I have used. Also the bounded bfgs is useful for many cases to enforce constraints on say pca.... Note that right now the version is not blas optimized. With jblas or netlib-java discussions that's going on it can be improved. Also it runs on a single thread which can be improved...so there is scope for further improvements in the code. Basically Xiangrui, is there a push back on making optimizers part of spark mllib ? I am exploring cg and qp solvers for spark mllib as well and I am developing these as part of mllib optimization. I was hoping we should be able to publish mllib as a maven artifact later. Thanks. Deb On Feb 25, 2014 6:53 AM, "Debasish Das" <debasish.da...@gmail.com> wrote: > Hi DB, Xiangrui, > > Mallet from cmu also has bfgs cg and a good optimization package. Do you > know if cpl license si > On Feb 22, 2014 11:50 AM, "Xiangrui Meng" <men...@gmail.com> wrote: > >> Hi DB, >> >> It is great to have the L-BFGS optimizer in MLlib and thank you for taking >> care of the license issue. I looked through your PR briefly. It contains a >> Java translation of the L-BFGS implementation, which is part of the RISO >> package. Is it possible that we ask its author to make a release on maven >> central and then we add it as a dependency instead of including the code >> directly? >> >> Best, >> Xiangrui >> >> >> On Sat, Feb 22, 2014 at 4:28 AM, DB Tsai <dbt...@alpinenow.com> wrote: >> >> > Hi guys, >> > >> > First of all, we would like to thank all the Spark community for >> > building such great platform for big data processing. We built the >> > multinomial logistic regression with LBFGS optimizer in Spark, and >> > LBFGS is a limited memory version of quasi-newton method which allows >> > us to train a very high-dimensional data without computing the Hessian >> > matrix as newton method required. >> > >> > In Strata Conference, we did a great demo using Spark with our MLOR to >> > train mnist8m dataset. We're able to train the model in 5 mins with 50 >> > iterations, and get 86% accuracy. The first iteration takes 19.8s, and >> > the remaining iterations take about 5~7s. >> > >> > We did comparison between LBFGS and SGD, and often we saw 10x less >> > steps in LBFGS while the cost of per step is the same (just computing >> > the gradient). >> > >> > The following is the paper by Prof. Ng at Stanford comparing different >> > optimizers including LBFGS and SGD. They use them in the context of >> > deep learning, but worth as reference. >> > >> http://cs.stanford.edu/~jngiam/papers/LeNgiamCoatesLahiriProchnowNg2011.pdf >> > >> > We would like to break our MLOR with LBFGS into three patches to >> > contribute to the community. >> > >> > 1) LBFGS optimizer - which can be used in logistic regression, and >> > liner regression or replacing any algorithms using SGD. >> > The core underneath LBFGS Java implementation we used is from RISO >> > project, and the author, Robert is so kind to relicense it to GPL and >> > Apache2 dual license. >> > >> > We're almost ready to submit a PR for LBFGS, see our github fork, >> > https://github.com/AlpineNow/incubator-spark/commits/dbtsai-LBFGS >> > >> > However, we don't use Updater in LBFGS since it designs for GD, and >> > for LBFGS, we don't need stepSize, and adaptive learning rate, etc. >> > While it seems to be difficult to fit the LBFGS updater logic (well, >> > in lbfgs library, the new weights is returned given old weights, loss, >> > and gradient) into the current framework, I was thinking to abstract >> > out the code computing the gradient and loss terms of regularization >> > into different place so that different optimizer can also use it. Any >> > suggestion about this? >> > >> > 2) and 3), we will add the MLOR gradient to MLLib, and add a few >> > examples. Finally, we will have some tweak using mapPartition instead >> > of map to further improve the performance. >> > >> > Thanks. >> > >> > Sincerely, >> > >> > DB Tsai >> > Machine Learning Engineer >> > Alpine Data Labs >> > -------------------------------------- >> > Web: http://alpinenow.com/ >> > >> >
