Re: Help alleviating OOM errors

Hi Yana, In 0.9.1, spark.shuffle.spill is set to true by default so you shouldn't need to manually set it. Here are a few common causes of OOMs in Spark: - Too few partitions: if one partition is too big, it may cause an OOM if there is not enough space to unroll the entire partition in memory.

Re: Help alleviating OOM errors

Can you elaborate why "You need to configure the spark.shuffle.spill true again in the config" -- the default for spark.shuffle.spill is set to true according to the doc(https://spark.apache.org/docs/0.9.1/configuration.html)? On OOM the tasks were process_local, which I understand is "as good as

Re: Help alleviating OOM errors

Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi On Mon, Jun 30, 2014 at 8:09 PM, Yana Kadiyska wrote: > Hi, > > our cluster seems to have a really hard time with OOM errors on the > executor. Periodically we'd see a task th