Hi Deb, CPL 1.0 is compatible if the inclusion is appropriately labeled (https://www.apache.org/legal/3party.html). I think it is great to have an L-BFGS optimizer in mllib, but we need to investigate some time to figure out which one to use. I'm not sure whether jblas or netlib-java will make a big difference here, because L-BFGS doesn't have dense level-2 BLAS operations. But it would be great if someone can do the comparison and show some numbers. FYI, spark-mllib is published in maven central.
Best, Xiangrui On Tue, Feb 25, 2014 at 7:07 AM, Debasish Das <debasish.da...@gmail.com> wrote: > Continuation on last email sent by mistake: > > Is cpl license is compatible with apache ? > > http://opensource.org/licenses/cpl1.0.php > > Mallet jars are available on maven. They have hessian based solvers which > looked interesting along with bfgs and cg. > > Definitely the lbfgs f2j looks promising as the base fortran code is > perhaps the best bfgs code I have used. Also the bounded bfgs is useful for > many cases to enforce constraints on say pca.... > > Note that right now the version is not blas optimized. With jblas or > netlib-java discussions that's going on it can be improved. Also it runs on > a single thread which can be improved...so there is scope for further > improvements in the code. > > Basically Xiangrui, is there a push back on making optimizers part of spark > mllib ? I am exploring cg and qp solvers for spark mllib as well and I am > developing these as part of mllib optimization. I was hoping we should be > able to publish mllib as a maven artifact later. > > Thanks. > Deb > On Feb 25, 2014 6:53 AM, "Debasish Das" <debasish.da...@gmail.com> wrote: > >> Hi DB, Xiangrui, >> >> Mallet from cmu also has bfgs cg and a good optimization package. Do you >> know if cpl license si >> On Feb 22, 2014 11:50 AM, "Xiangrui Meng" <men...@gmail.com> wrote: >> >>> Hi DB, >>> >>> It is great to have the L-BFGS optimizer in MLlib and thank you for taking >>> care of the license issue. I looked through your PR briefly. It contains a >>> Java translation of the L-BFGS implementation, which is part of the RISO >>> package. Is it possible that we ask its author to make a release on maven >>> central and then we add it as a dependency instead of including the code >>> directly? >>> >>> Best, >>> Xiangrui >>> >>> >>> On Sat, Feb 22, 2014 at 4:28 AM, DB Tsai <dbt...@alpinenow.com> wrote: >>> >>> > Hi guys, >>> > >>> > First of all, we would like to thank all the Spark community for >>> > building such great platform for big data processing. We built the >>> > multinomial logistic regression with LBFGS optimizer in Spark, and >>> > LBFGS is a limited memory version of quasi-newton method which allows >>> > us to train a very high-dimensional data without computing the Hessian >>> > matrix as newton method required. >>> > >>> > In Strata Conference, we did a great demo using Spark with our MLOR to >>> > train mnist8m dataset. We're able to train the model in 5 mins with 50 >>> > iterations, and get 86% accuracy. The first iteration takes 19.8s, and >>> > the remaining iterations take about 5~7s. >>> > >>> > We did comparison between LBFGS and SGD, and often we saw 10x less >>> > steps in LBFGS while the cost of per step is the same (just computing >>> > the gradient). >>> > >>> > The following is the paper by Prof. Ng at Stanford comparing different >>> > optimizers including LBFGS and SGD. They use them in the context of >>> > deep learning, but worth as reference. >>> > >>> http://cs.stanford.edu/~jngiam/papers/LeNgiamCoatesLahiriProchnowNg2011.pdf >>> > >>> > We would like to break our MLOR with LBFGS into three patches to >>> > contribute to the community. >>> > >>> > 1) LBFGS optimizer - which can be used in logistic regression, and >>> > liner regression or replacing any algorithms using SGD. >>> > The core underneath LBFGS Java implementation we used is from RISO >>> > project, and the author, Robert is so kind to relicense it to GPL and >>> > Apache2 dual license. >>> > >>> > We're almost ready to submit a PR for LBFGS, see our github fork, >>> > https://github.com/AlpineNow/incubator-spark/commits/dbtsai-LBFGS >>> > >>> > However, we don't use Updater in LBFGS since it designs for GD, and >>> > for LBFGS, we don't need stepSize, and adaptive learning rate, etc. >>> > While it seems to be difficult to fit the LBFGS updater logic (well, >>> > in lbfgs library, the new weights is returned given old weights, loss, >>> > and gradient) into the current framework, I was thinking to abstract >>> > out the code computing the gradient and loss terms of regularization >>> > into different place so that different optimizer can also use it. Any >>> > suggestion about this? >>> > >>> > 2) and 3), we will add the MLOR gradient to MLLib, and add a few >>> > examples. Finally, we will have some tweak using mapPartition instead >>> > of map to further improve the performance. >>> > >>> > Thanks. >>> > >>> > Sincerely, >>> > >>> > DB Tsai >>> > Machine Learning Engineer >>> > Alpine Data Labs >>> > -------------------------------------- >>> > Web: http://alpinenow.com/ >>> > >>> >>
