I am having some issues with processes running out of memory and I'm wondering if I'm setting things up incorrectly.
I am running a job on two nodes with 24 cores and 256Gb of memory each. I start the pyspark shell with SPARK_EXECUTOR_MEMORY=210gb. When I run the job with anything more than 8 cores, the processes start dying off with out of memory errors. But when I watch the memory consumption using top on the two execute nodes, the individual processes never seem to exceed the per-core memory and the nodes themselves are far from running out of memory. So I'm wondering if Spark is setting the per-core memory limit somewhere? Thanks, Rok --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
