Have you tried periodic checkpoints ?
Cheers
> On Aug 16, 2016, at 5:50 AM, pseudo oduesp wrote:
>
> Hi ,
> how we can deal after raise stackoverflow trigger by long lineage ?
> i mean i have this error and how resolve it wiyhout creating new se
Hi ,
how we can deal after raise stackoverflow trigger by long lineage ?
i mean i have this error and how resolve it wiyhout creating new session
thanks
tried using sc.saveAsObjectFile per iteration and then reading from
> file (sc.objectFile) during the next iteration. Unfortunately I noticed
> that
> the folders are created per iteration are increasing while I was expecting
> from them to have equal size per iteration.
>
> pleas
.com/Stack-overflow-error-caused-by-long-lineage-RDD-created-after-many-recursions-tp25240.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For ad
...@gmail.com]
Sent: Friday, May 8, 2015 2:51 PM
To: Shao, Saisai
Cc: user@spark.apache.org
Subject: Re: Possible long lineage issue when using DStream to update a normal
RDD
Thank you for this suggestion! But may I ask what's the advantage to use
checkpoint instead of cache here? Cuz they both cut li
DD`, I have a
> similar scenario and I use checkpoint to workaround this problem :)
>
> Thanks
> Jerry
>
> -Original Message-
> From: yaochunnan [mailto:yaochun...@gmail.com]
> Sent: Friday, May 8, 2015 1:57 PM
> To: user@spark.apache.org
> Subject: Possible long line
Subject: Possible long lineage issue when using DStream to update a normal RDD
Hi all,
Recently in our project, we need to update a RDD using data regularly received
from DStream, I plan to use "foreachRDD" API to achieve this:
var MyRDD = ...
dstream.foreachRDD { rdd =>
MyRDD = M
y appreciated.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Possible-long-lineage-issue-when-using-DStream-to-update-a-normal-RDD-tp22812.html
Sent from the Apache Spark User List mailing list archive
