About `LogAndThresholdExceptionHandler`: If the handler needs to keep track of number of failed messages, than it becomes stateful -- not sure if we should do that. But maybe we can introduce 2 metrics (might be an interesting metric to report to the user anyway) and allow programmatic access to this metric in the handler. This might also work better, as we should use "windowed" failure rate anyway -- an absolute count of failed messages from beginning of time would not work very well IMHO.
About "dead letter queue": Yes, that what my thought -- we provide an out-of-the-box handler that writes to a topic (thus, this part is covered by setting the corresponding handler in the config). Thus, not sure if we need a config for the topic name as we could exploit `Configurable` or we just use a fixed name like "<application.id>-dead-letters" (or some better name). Not sure how important it is to configure the name (the less config the better -- otherwise it's getting to hard to see what is important/relevant and what not) -Matthias On 5/26/17 2:13 AM, Eno Thereska wrote: > Replying to Avi's and Matthias' questions in one go inline: > >> On 25 May 2017, at 19:27, Matthias J. Sax <matth...@confluent.io> wrote: >> >> Thanks for the KIP Eno! >> >> Couple of comments: >> >> I think we don't need `RecordContext` in `RecordExceptionHandler#handle` >> because the `ConsumerRecord` provides all this information anyway. > > Good point, fixed. > >> >> Why we introduce `ExceptionType` and not just hand in the actual exception? >> > > Ok, changed. > >> As return type of `handle()` is void, how would the handler fail? By >> throwing an exception? Maybe it would be better to add a proper return >> type from the beginning on -- this might also make backward >> compatibility easier later on. > > Added two options to the KIP, see what you think (Continue/Fail). If we add > Retry later on, not sure if that'd trigger a KIP. > >> >> Question about `LogAndThresholdExceptionHandler` -- how would we be able >> to track this? > > I'm thinking StreamsThread instantiates one such handler that keeps a count > of failed messages. The handler will need to know the total of successful > messages too though, and come to think of it some more, we don't have a > metric that keeps that. We can either add that metric (in JMX) or > alternatively change the threshold handler to respond to number of failures, > instead of %. Preferences? >> >> >> With regard to `DeserializationException`, do you thing it might make >> sense to have a "dead letter queue" as a feature to provide out-of-the-box? > > We could provide a special topic where bad messages go to, and then we'd have > to add a config option for the user to provide a topic. Is that what you're > thinking? > > Eno > > >> >> >> -Matthias >> >> On 5/25/17 2:47 AM, Eno Thereska wrote: >>> Hi there, >>> >>> I’ve added a KIP on improving exception handling in streams: >>> KIP-161: streams record processing exception handlers. >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+record+processing+exception+handlers >>> >>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-161:+streams+record+processing+exception+handlers> >>> >>> Discussion and feedback is welcome, thank you. >>> Eno >>> >> >
signature.asc
Description: OpenPGP digital signature
