Re: Looking for help about stackoverflow in spark

2016-06-30 Thread Chanh Le
dded documents in > it.) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Looking-for-help-about-stackoverflow-in-spark-tp27255.html > Sent from the Apache Spark User List mailing list archive

Looking for help about stackoverflow in spark

2016-06-30 Thread johnzeng
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Looking-for-help-about-stackoverflow-in-spark-tp27255.html Sent from the Apache Spark User List mailing list archive at Nabble.com. -

Re: StackOverflow in Spark

2016-06-13 Thread Terry Hoo
6 > *Aan:* Yash Sharma > *CC:* Ted Yu ; Matthew Young ; > Michel Hubert ; user@spark.apache.org > *Onderwerp:* Re: StackOverflow in Spark > > > > Stackoverflow is generated when DAG is too log as there are many > transformations in lot of iterations. Please use checkpointing to

RE: StackOverflow in Spark

2016-06-13 Thread Michel Hubert
@spark.apache.org Onderwerp: Re: StackOverflow in Spark Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and break the linage to get away from this stack overflow error. Look into checkpoint fuction. Thanks Hope

Re: StackOverflow in Spark

2016-06-01 Thread Rishabh Wadhawan
Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and break the linage to get away from this stack overflow error. Look into checkpoint fuction. Thanks Hope it helps. Let me know if you need anymore he

Re: StackOverflow in Spark

2016-06-01 Thread Yash Sharma
Not sure if its related, But I got a similar stack overflow error some time back while reading files and converting them to parquet. > Stack trace- > 16/06/02 02:23:54 INFO YarnAllocator: Driver requested a total number of > 32769 executor(s). > 16/06/02 02:23:54 INFO ExecutorAllocationManager:

Re: StackOverflow in Spark

2016-06-01 Thread Ted Yu
Looking at Michel's stack trace, it seems to be different issue. > On Jun 1, 2016, at 7:45 PM, Matthew Young wrote: > > Hi, > > It's related to the one fixed bug in Spark, jira ticket SPARK-6847 > > Matthew Yang > >> On Wed, May 25, 2016 at 7:48 PM, Michel Hubert wrote: >> >> >> Hi, >>

Re: StackOverflow in Spark

2016-06-01 Thread Matthew Young
Hi, It's related to the one fixed bug in Spark, jira ticket SPARK-6847 Matthew Yang On Wed, May 25, 2016 at 7:48 PM, Michel Hubert wrote: > > > Hi, > > > > > > I have an Spark application which generates StackOverflowError exceptions > after 3

StackOverflow in Spark

2016-05-25 Thread Michel Hubert
Hi, I have an Spark application which generates StackOverflowError exceptions after 30+ min. Anyone any ideas? Seems like problems with deserialization of checkpoint data? 16/05/25 10:48:51 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 55449.0 (TID 5584, host81440-cld.opentsp.co