Hi, I have upgraded 5 node spark cluster from spark-1.5 to spark-1.6 (to use mapWithState function). After using spark-1.6, I am getting a strange behaviour of spark, jobs are not using multiple executors of different nodes at a time means there is no parallel processing if each node having single worker and executor. I am running jobs in spark standalone mode.
I observed following points related to this issue. 1. If I run same job with spark-1.5 then this will use multiple executors across different nodes at a time. 2. In Spark-1.6, If I increase no of cores(spark.cores.max) then jobs are running in parallel thread but within same executor. 3. In Spark-1.6, If I increase no of worker instances on each node then jobs are running in parallel as no of workers but within same executor. Can anyone suggest, why spark 1.6 can not use multiple executors across different node at a time for parallel processing. Your suggestion will be highly appreciated. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-executors-in-spark-1-6-and-spark-1-5-tp26733.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
