Re: Caching intermediate results in Spark ML pipeline?

/main/scala/org/apache/spark/ml/tuning/CrossValidator.scala#L85> > . > > Otherwise, perhaps you can elaborate more on your particular use case for > caching intermediate results and if the current API doesn't support it we > can create a JIRA for it. > > On Tue, Sep 15,

Re: Caching intermediate results in Spark ML pipeline?

iple runs, you have a few > options: > * cache it first and pass it in as an argument to searchRun > * use a creational pattern like singleton to ensure only one instantiation > > On Tue, Sep 15, 2015 at 12:49 AM, Jingchu Liu > wrote: > >> Hey Feynman, >> >>

Re: Caching intermediate results in Spark ML pipeline?

sponse, this will still require you call each > PipelineStage's `transform` method (i.e. to NOT use the overall > Pipeline.setStages API) > > On Mon, Sep 14, 2015 at 10:45 PM, Jingchu Liu > wrote: > >> Hey Feynman, >> >> Thanks for your response, but I&#x

Re: Caching intermediate results in Spark ML pipeline?

API). See associated > JIRAs <https://issues.apache.org/jira/browse/SPARK-4587>. > > Pipeline persistence is on the 1.6 roadmap, JIRA here > <https://issues.apache.org/jira/browse/SPARK-6725>. > > Feynman > > On Mon, Sep 14, 2015 at 9:20 PM, Jingchu Liu wrote: > >&g

Caching intermediate results in Spark ML pipeline?

Hi all, I have a question regarding the ability of ML pipeline to cache intermediate results. I've posted this question on stackoverflow but got no answer, hope someone here can help me out. ==