Re: OutOfMemory with wide (289 column) dataframe

2016-04-01 Thread Ted Yu
bq. This was a big help! The email (maybe only addressed to you) didn't come with your latest reply. Do you mind sharing it ? Thanks On Fri, Apr 1, 2016 at 11:37 AM, ludflu wrote: > This was a big help! For the benefit of my fellow travelers running spark > on > EMR: > > I made a json file wi

Re: OutOfMemory with wide (289 column) dataframe

2016-04-01 Thread ludflu
This was a big help! For the benefit of my fellow travelers running spark on EMR: I made a json file with the following: [ { "Classification": "yarn-site", "Properties": { "yarn.nodemanager.pmem-check-enabled": "false", "yarn.nodemanager.vmem-check-enabled": "false" } } ] and then I created my c