How to reuse a ML trained model?

Hi, I checked a few ML algorithms in MLLib. https://spark.apache.org/docs/0.8.1/api/mllib/index.html#org.apache.spark.mllib.classification.LogisticRegressionModel I could not find a way to save the trained model. Does this means I have to train my model every time? Is there a more economic way t

Re: How to reuse a ML trained model?

You can serialize your trained model to persist somewhere. Ekrem Aksoy On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen wrote: > Hi, > > I checked a few ML algorithms in MLLib. > > > https://spark.apache.org/docs/0.8.1/api/mllib/index.html#org.apache.spark.mllib.classification.LogisticRegressionModel >

Re: How to reuse a ML trained model?

Ah~it is serializable. Thanks! On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy wrote: > You can serialize your trained model to persist somewhere. > > Ekrem Aksoy > > On Sat, Mar 7, 2015 at 12:10 PM, Xi Shen wrote: > >> Hi, >> >> I checked a few ML algorithms in MLLib. >> >> >> https://spark.apach

Re: How to reuse a ML trained model?

Wait...it seem SparkContext does not provide a way to save/load object files. It can only save/load RDD. What do I missed here? Thanks, David On Sat, Mar 7, 2015 at 11:05 PM Xi Shen wrote: > Ah~it is serializable. Thanks! > > > On Sat, Mar 7, 2015 at 10:59 PM Ekrem Aksoy wrote: > >> You can

Re: Spark streaming and executor object reusage

In the example with "createNewConnection()", a connection is created for every partition of every batch of input. You could take the idea further and share connections across partitions or batches. This requires them to have a lifecycle beyond foreachRDD. That's accomplishable with some kind of sta

Re: Spark streaming and executor object reusage

Thanks Sean this is really helpful. Please see comments line. Sent from my iPad > On Mar 7, 2015, at 4:45 AM, Sean Owen wrote: > > In the example with "createNewConnection()", a connection is created > for every partition of every batch of input. You could take the idea > further and share conn

Re: Spark streaming and executor object reusage

On Sat, Mar 7, 2015 at 4:17 PM, Jean-Pascal Billaud wrote: > So given this let's go a bit further. Imagine my static factory provides a > stats collector that my various map() code would use to export some metrics > while mapping tuples. This stats collector comes with a timer that flush the >

Re: Spark streaming and executor object reusage

Thanks a lot. Sent from my iPad > On Mar 7, 2015, at 8:26 AM, Sean Owen wrote: > >> On Sat, Mar 7, 2015 at 4:17 PM, Jean-Pascal Billaud >> wrote: >> So given this let's go a bit further. Imagine my static factory provides a >> stats collector that my various map() code would use to export so

Re: How to reuse a ML trained model?

Hi, There is model import/export for some of the ML algorithms on the current master (and they'll be shipped with the 1.3 release). Burak On Mar 7, 2015 4:17 AM, "Xi Shen" wrote: > Wait...it seem SparkContext does not provide a way to save/load object > files. It can only save/load RDD. What do

MLlib/kmeans newbie question(s)

Hi all, I'm very new to machine learning algorithms and Spark. I'm follow the Twitter Streaming Language Classifier found here: http://databricks.gitbooks.io/databricks-spark-reference-applications/content/twitter_classifier/README.html Specifically this code: http://databricks.gitbooks.io/data

Re: distcp on ec2 standalone spark cluster

Did you get this to work? I got pass the issues with the cluster not startetd problem I am having problem where distcp with s3 URI says incorrect forlder path and s3n:// hangs. stuck for 2 days :( Thanks -R -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/dis

Re: Connection PHP application to Spark Sql thrift server

Sorry for late reply. I have tried to connect to the hive server instead of the spark sql but the same exception is thrown in the hive server logs. The only difference is the hive log has a little more information than the spark sql logs. The hive server logs has this message "TTransportException

Bulk insert strategy

While processing DStream in the Spark Programming Guide, the suggested usage of connection is the following, dstream.foreachRDD(rdd => { rdd.foreachPartition(partitionOfRecords => { // ConnectionPool is a static, lazily initialized pool of connections val connection = Co