Re: Spark on YARN memory consumption

2016-03-11 Thread Jan Štěrba
těrba > Sent: Friday, March 11, 2016 8:27 AM > To: User > Subject: Spark on YARN memory consumption > > > > Hello, > > I am exprimenting with tuning an on demand spark-cluster on top of our > cloudera hadoop. I am running Cloudera 5.5.2 with Spark 1.5 right now > and

RE: Spark on YARN memory consumption

2016-03-11 Thread Silvio Fiorito
<mailto:i...@jansterba.com> Sent: Friday, March 11, 2016 8:27 AM To: User<mailto:user@spark.apache.org> Subject: Spark on YARN memory consumption Hello, I am exprimenting with tuning an on demand spark-cluster on top of our cloudera hadoop. I am running Cloudera 5.5.2 with Spark 1

Spark on YARN memory consumption

2016-03-11 Thread Jan Štěrba
Hello, I am exprimenting with tuning an on demand spark-cluster on top of our cloudera hadoop. I am running Cloudera 5.5.2 with Spark 1.5 right now and I am running spark in yarn-client mode. Right now my main experimentation is about spark.executor.memory property and I have noticed a strange be