Re: StackOverflow in Spark

6 > *Aan:* Yash Sharma > *CC:* Ted Yu ; Matthew Young ; > Michel Hubert ; user@spark.apache.org > *Onderwerp:* Re: StackOverflow in Spark > > > > Stackoverflow is generated when DAG is too log as there are many > transformations in lot of iterations. Please use checkpointing to

RE: StackOverflow in Spark

@spark.apache.org Onderwerp: Re: StackOverflow in Spark Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and break the linage to get away from this stack overflow error. Look into checkpoint fuction. Thanks Hope

Re: StackOverflow in Spark

Stackoverflow is generated when DAG is too log as there are many transformations in lot of iterations. Please use checkpointing to store the DAG and break the linage to get away from this stack overflow error. Look into checkpoint fuction. Thanks Hope it helps. Let me know if you need anymore he

Re: StackOverflow in Spark

Not sure if its related, But I got a similar stack overflow error some time back while reading files and converting them to parquet. > Stack trace- > 16/06/02 02:23:54 INFO YarnAllocator: Driver requested a total number of > 32769 executor(s). > 16/06/02 02:23:54 INFO ExecutorAllocationManager:

Re: StackOverflow in Spark

Looking at Michel's stack trace, it seems to be different issue. > On Jun 1, 2016, at 7:45 PM, Matthew Young wrote: > > Hi, > > It's related to the one fixed bug in Spark, jira ticket SPARK-6847 > > Matthew Yang > >> On Wed, May 25, 2016 at 7:48 PM, Michel Hubert wrote: >> >> >> Hi, >>

Re: StackOverflow in Spark

Hi, It's related to the one fixed bug in Spark, jira ticket SPARK-6847 Matthew Yang On Wed, May 25, 2016 at 7:48 PM, Michel Hubert wrote: > > > Hi, > > > > > > I have an Spark application which generates StackOverflowError exceptions > after 3