I add this code in foreachRDD block . ``` rdd.persist(StorageLevel.MEMORY_AND_DISK) ```
This exception no occur agein.But many executor dead showing in spark streaming UI . ``` User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 21 in stage 1194.0 failed 4 times, most recent failure: Lost task 21.3 in stage 1194.0 (TID 2475, 2.dev3, executor 66): ExecutorLostFailure (executor 66 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits. 3.5 GB of 3.5 GB physical memory used. Consider boosting spark.yarn.executor.memoryOverhead. ``` -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-exectors-memory-increasing-and-executor-killed-by-yarn-tp28500p28506.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
