Re: Memory problems when calling pipe()

Hi, I finally solved the problem by setting spark.yarn.executor.memoryOverhead with the option --conf "spark.yarn.executor.memoryOverhead=" for spark-submit, as pointed out in http://stackoverflow.com/questions/28404714/yarn-why-doesnt-task-go-out-of-heap-space-but-container-gets-killed and ht

Memory problems when calling pipe()

Hi, I'm having problems using pipe() from a Spark program written in Java, where I call a python script, running in a YARN cluster. The problem is that the job fails when YARN kills the container because the python script is going beyond the memory limits. I get something like this in the log: 0