Please see
[SPARK-13465] Add a task failure listener to TaskContext
On Sat, Dec 19, 2015 at 3:44 PM, Neelesh wrote:
> Hi,
> I'm trying to build automatic Kafka watermark handling in my stream apps
> by overriding the KafkaRDDIterator, and adding a taskcompletionlistener and
> updating watermar
I also created a JIRA for task failures
https://issues.apache.org/jira/browse/SPARK-12452
On Mon, Dec 21, 2015 at 9:54 AM, Neelesh wrote:
> I am leaning towards something like that. Things get interesting when
> multiple different transformations and regrouping happen. At the end of it
> all, wh
I am leaning towards something like that. Things get interesting when
multiple different transformations and regrouping happen. At the end of it
all, when the "task" is done, we no longer are sure which kafka partition
they came from, even when all transforms/ grouping happen local to the
original
Honestly it's a lot easier to deal with this using transactions.
Someone else would have to speak to the possibility of getting task
failures added to listener callbacks.
On Sat, Dec 19, 2015 at 5:44 PM, Neelesh wrote:
> Hi,
> I'm trying to build automatic Kafka watermark handling in my strea
Hi,
I'm trying to build automatic Kafka watermark handling in my stream apps
by overriding the KafkaRDDIterator, and adding a taskcompletionlistener and
updating watermarks if task was completed (the iterator has access to
offsets). But I found out that there is no way to listen to a task error
i
