On Thu, Mar 6, 2014 at 4:26 PM, David Hall <d...@cs.berkeley.edu> wrote: > I'm not sure why Spark should be serializing LBFGS? Shouldn't it live on > the controller node? Or is this a per-node thing? > > But no problem to make it serializable. It will live in the controller node. Only RDD operations are per-node thing. I'm calling RDD operations inside the DiffFunction, so Spark should not serialize anything. But it seems that Spark serialize evey parent objects; can Spark anyone confirm this?
> Err, yeah. I should probably have LBFGS do this automatically, but there's > a CachedDiffFunction that gets rid of the redundant calculations. It will nice to have LBFHS do this automatically. Just try CachedDiffFunction, and it works. Sincerely, DB Tsai Machine Learning Engineer Alpine Data Labs -------------------------------------- Web: http://alpinenow.com/
