Guozhang, I read the comments in KAFKA-6376 and realized that we do something similar to handle exception in process(). We register a custom metric "process-skipped" and our alerting system looks for both this and skipped-records.
Srikanth On Fri, Jan 26, 2018 at 10:26 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Helo Srikanth, > > Thanks for reporting this, as I checked the code > skippedDueToDeserializationError is effectively only recorded when the > DeserializationHandlerResponse is not set to FAIL. I agree it is not > exactly matching the documentation's guidance, and will try to file a JIRA > and fix it. > > As for skippedDueToDeserializationError and skipped-records-rate, there is > an open JIRA discussing about this: > https://issues.apache.org/jira/browse/KAFKA-6376, just FYI. > > > Could you share on which version of Kafka did you observe this issue? > > Guozhang > > On Fri, Jan 26, 2018 at 6:30 AM, Srikanth <srikanth...@gmail.com> wrote: > > > Hello, > > > > As per doc when LogAndContinueExceptionHandler is used it will set > > skippedDueToDeserializationError-rate metric to indicate deserialization > > error. > > I notice that it is never set. Instead skipped-records-rate is set. My > > understanding was that skipped-records-rate is set due to timestamp > > extraction errors. > > > > Ex, I sent a few invalid records to a topic and was able to see exception > > during deserialization. > > > > org.apache.kafka.common.errors.SerializationException: Error > deserializing > > Avro message for id -1 > > Caused by: org.apache.kafka.common.errors.SerializationException: > Unknown > > magic byte! > > 18/01/24 06:50:09 WARN StreamThread: Exception caught during > > Deserialization, taskId: 0_0, topic: docker.event.1, partition: 0, > offset: > > 3764 > > > > These incremented skipped-records-[rate|total]. > > > > Thanks, > > Srikanth > > > > > > -- > -- Guozhang >
