Re: Open source Spark based projects

2016-09-25 Thread Simon Chan
PredictionIO is an open-source machine learning server project based on Spark - http://predictionio.incubator.apache.org/ Simon On Fri, Sep 23, 2016 at 12:46 PM, manasdebashiskar wrote: > check out spark packages https://spark-packages.org/ and you will find few > awesome and a lot of super aw

Re: How to reuse a ML trained model?

2015-03-08 Thread Simon Chan
You may also take a look at PredictionIO, which can persist and then deploy MLlib models as web services. Simon On Sunday, March 8, 2015, Sean Owen wrote: > You dont need SparkContext to simply serialize and deserialize objects. It > is Java mechanism. > On Mar 8, 2015 10:29 AM, "Xi Shen" > wr

Re: deploying a model built in mllib

2014-11-04 Thread Simon Chan
The latest version of PredictionIO, which is now under Apache 2 license, supports the deployment of MLlib models on production. The "engine" you build will including a few components, such as: - Data - includes Data Source and Data Preparator - Algorithm(s) - Serving I believe that you can do the

MLlib and PredictionIO sample code

2014-11-04 Thread Simon Chan
Hey guys, I have written a tutorial on deploying MLlib's models on production with open source PredictionIO: http://docs.prediction.io/0.8.1/templates/ The goal is to add the following features to MLlib, with production application in mind: - JSON query to retrieve prediction online - Separation-

Re: word2vec: how to save an mllib model and reload it?

2014-11-07 Thread Simon Chan
Just want to elaborate more on Duy's suggestion on using PredictionIO. PredictionIO will store the model automatically if you return it in the training function. An example using CF: def train(data: PreparedData): PersistentMatrixFactorizationModel = { val m = ALS.train(data.ratings, ap.rank

Re: MLLib in Production

2014-12-10 Thread Simon Chan
Hi Klaus, PredictionIO is an open source product based on Spark MLlib for exactly this purpose. This is the tutorial for classification in particular: http://docs.prediction.io/classification/quickstart/ You can add custom serving logics and retrieve prediction result through REST API/SDKs at oth

Re: Launching Spark app in client mode for standalone cluster

2015-01-04 Thread Simon Chan
Boromir, You may like to take a look at how we make Spray and Spark working together at the PredictionIO project: https://github.com/PredictionIO/PredictionIO Simon On Sun, Jan 4, 2015 at 8:31 PM, Chester At Work wrote: > Just a guess here, may not be correct. > > Spray needs to start akka

Re: Spark for core business-logic? - Replacing: MongoDB?

2015-01-05 Thread Simon Chan
Alec, If you are looking for a Machine Learning stack that supports business-logics, you may take a look at PredictionIO: http://prediction.io/ It's based on Spark and HBase. Simon On Mon, Jan 5, 2015 at 6:14 PM, Alec Taylor wrote: > Thanks all. To answer your clarification questions: > > -

Re: Spark for core business-logic? - Replacing: MongoDB?

2015-01-05 Thread Simon Chan
ult computations). > > However, does it handle the actual data storage? - E.g.: CRUD documents > > On Tue, Jan 6, 2015 at 1:18 PM, Simon Chan wrote: > > Alec, > > > > If you are looking for a Machine Learning stack that supports > > business-logics, you may t

Re: Is there any Spark implementation for Item-based Collaborative Filtering?

2015-01-12 Thread Simon Chan
Also a ready-to-use server with Spark MLlib: http://docs.prediction.io/recommendation/quickstart/ The source code is here: https://github.com/PredictionIO/PredictionIO/tree/develop/templates/scala-parallel-recommendation Simon On Sun, Nov 30, 2014 at 12:17 PM, Pat Ferrel wrote: > Actually the