Please see [SPARK-13465] Add a task failure listener to TaskContext On Sat, Dec 19, 2015 at 3:44 PM, Neelesh <neele...@gmail.com> wrote:
> Hi, > I'm trying to build automatic Kafka watermark handling in my stream apps > by overriding the KafkaRDDIterator, and adding a taskcompletionlistener and > updating watermarks if task was completed (the iterator has access to > offsets). But I found out that there is no way to listen to a task error > inside the executors. Only the driver gets the TaskReason notification, but > not the task completion listener. This basically means watermarks will be > updated regardless of whether the task completed successfully or not. > > Is there any way to listen for the task failures on the executors? > > Thanks! > -neelesh > >
