Hi,
Thanks for the KIP.
I wonder whether this idea would be better implemented using a new method on the
SinkTaskContext.
public void putFailed(Collection<SinkRecord> records)
Then the rest of KIP-298 could apply. Failed records could be put to the DLQ or
logged, as
appropriate. I think there's real value in keeping the rest of the error
handling intact and using
the same mechanism for failed puts as for failures in the other stages of
processing. If you want
to retry a batch, throw a RetriableException. If the record cannot be processed
perhaps due to
a size limit in the target system, flag it as failed and the framework can
invoke its error handling
code on the offending record or records. The only stipulation is that the error
action needs to be
completed before the offset for a failed record is committed or it might be
lost.
Thanks,
Andrew
On 07/05/2020, 19:09, "Aakash Shah" <as...@confluent.io> wrote:
Hello all,
I've created a KIP to handle error reporting for records in sink
connectors, specifically within the context of put(...):
https://cwiki.apache.org/confluence/display/KAFKA/KIP-610%3A+Error+Reporting+in+Sink+Connectors
I would appreciate any kind of feedback.
Thanks,
Aakash
