Hi, I am wondering about the safety of the *SparkContext#cancelJobGroup* method that should allow to stop specific (ie not all) jobs inside a spark context.
There is a big disclaimer ( https://spark.apache.org/docs/1.6.0/api/java/org/apache/spark/SparkContext.html#setJobGroup(java.lang.String,%20java.lang.String,%20boolean) . If interruptOnCancel is set to true for the job group, then job > cancellation will result in Thread.interrupt() being called on the job's > executor threads. This is useful to help ensure that the tasks are actually > stopped in a timely manner, but is off by default due to HDFS-1208, where > HDFS may respond to Thread.interrupt() by marking nodes as dead. I have two main questions : 1. What is the expected behavior if it is not interrupted on cancel? I am especially curious about the YARN case with HDFS but any info is welcome. 2. Who is or was using the *interruptOnCancel* ? Do you got burn? It is still working without any incident? Thanks in advance for any info, feedbacks and war stories. Bertrand Dechoux
