: Akhil Das; dev
Subject: Re: Loading previously serialized object to Spark
Well, it is the standard "hacky" way for model save/load in MLlib. We have
SPARK-4587 and SPARK-5991 to provide save/load for all MLlib models, in an
exchangeable format. -Xiangrui
On Mon, Mar 9, 2015 at 12:25
: Ulanov, Alexander
> Cc: Akhil Das; dev
> Subject: Re: Loading previously serialized object to Spark
>
> Could you try `sc.objectFile` instead?
>
> sc.parallelize(Seq(model), 1).saveAsObjectFile("path") val sameModel =
> sc.objectFile[NaiveBayesModel]("path&qu
: Ulanov, Alexander
Cc: Akhil Das; dev
Subject: Re: Loading previously serialized object to Spark
Could you try `sc.objectFile` instead?
sc.parallelize(Seq(model), 1).saveAsObjectFile("path") val sameModel =
sc.objectFile[NaiveBayesModel]("path").first()
-Xiangrui
On Mon,
ternal Spark serializer:
> val serializer = SparkEnv.get.closureSerializer.newInstance
>
>
> -Original Message-
> From: Ulanov, Alexander
> Sent: Monday, March 09, 2015 10:37 AM
> To: Akhil Das
> Cc: dev
> Subject: RE: Loading previously serialized object to Spa
ClosureCleaner.scala:158)
at org.apache.spark.SparkContext.clean(SparkContext.scala:1453)
at org.apache.spark.rdd.RDD.map(RDD.scala:273)
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: Sunday, March 08, 2015 3:17 AM
To: Ulanov, Alexander
Cc: dev
Subject: Re: Loading previously serialize
ClosureCleaner.scala:158)
at org.apache.spark.SparkContext.clean(SparkContext.scala:1453)
at org.apache.spark.rdd.RDD.map(RDD.scala:273)
From: Akhil Das [mailto:ak...@sigmoidanalytics.com]
Sent: Sunday, March 08, 2015 3:17 AM
To: Ulanov, Alexander
Cc: dev
Subject: Re: Loading previously serialize
Can you paste the complete code?
Thanks
Best Regards
On Sat, Mar 7, 2015 at 2:25 AM, Ulanov, Alexander
wrote:
> Hi,
>
> I've implemented class MyClass in MLlib that does some operation on
> LabeledPoint. MyClass extends serializable, so I can map this operation on
> data of RDD[LabeledPoints],
Hi,
I've implemented class MyClass in MLlib that does some operation on
LabeledPoint. MyClass extends serializable, so I can map this operation on data
of RDD[LabeledPoints], such as data.map(lp => MyClass.operate(lp)). I write
this class in file with ObjectOutputStream.writeObject. Then I stop
