Honestly it's a lot easier to deal with this using transactions. Someone else would have to speak to the possibility of getting task failures added to listener callbacks.
On Sat, Dec 19, 2015 at 5:44 PM, Neelesh <neele...@gmail.com> wrote: > Hi, > I'm trying to build automatic Kafka watermark handling in my stream apps > by overriding the KafkaRDDIterator, and adding a taskcompletionlistener and > updating watermarks if task was completed (the iterator has access to > offsets). But I found out that there is no way to listen to a task error > inside the executors. Only the driver gets the TaskReason notification, but > not the task completion listener. This basically means watermarks will be > updated regardless of whether the task completed successfully or not. > > Is there any way to listen for the task failures on the executors? > > Thanks! > -neelesh >
