HI Zihan, Thanks for the KIP! I have some questions that I'm hoping we can address to help better understand the motivation for this proposal.
1. In the "Motivation" section it's written that "If users want to store their broken records, they have to config a broken record queue, which is too much work for them in some cases." Could you elaborate on what makes this a lot of work? Ideally, users should be able to configure the dead letter queue by specifying a value for the " errors.deadletterqueue.topic.name" property in their sink connector config; this doesn't seem like a lot of work on the surface. 2. If the "errors.tolerance" property is set to "continue", would sink connectors be able to differentiate between well-formed records whose successfully-deserialized contents are byte arrays and malformed records whose contents are the still-serialized byte arrays of the Kafka message from which they came? 3. I think it's somewhat implied by the KIP, but it'd be nice to see what the schema for a malformed record would be. Null? Byte array? Optional byte array? 4. This is somewhat covered by the first question, but it seems worth pointing out that this exact functionality can already be achieved by using features already provided by the framework. Configure your connector to send malformed records to a dead letter queue topic, and configure a separate connector to consume from that dead letter queue topic, use the ByteArrayConverter to deserialize records, and send those records to the destination sink. It'd be nice if this were called out in the "Rejected Alternatives" section with a reason on why the changes proposed in the KIP are preferable, especially since it may still work as a viable workaround for users who are working on older versions of the Connect framework. Looking forward to the discussion! Cheers, Chris On Tue, Mar 24, 2020 at 11:50 AM Zihan Li <lizi...@umich.edu> wrote: > Hi, > > I just want to re-up this discussion thread about KIP-582 Add a "continue" > option for Kafka Connect error handling. > > Wiki page: https://cwiki.apache.org/confluence/x/XRvcC < > https://cwiki.apache.org/confluence/x/XRvcC> > > JIRA: https://issues.apache.org/jira/browse/KAFKA-9740 < > https://issues.apache.org/jira/browse/KAFKA-9740> > > Please share your thoughts about adding this new error handling option to > Kafka Connect. > > Best, > Zihan > > > On Mar 18, 2020, at 12:55 PM, Zihan Li <lizi...@umich.edu> wrote: > > > > Hi all, > > > > I'd like to use this thread to discuss KIP-582 Add a "continue" option > for Kafka Connect error handling, please see detail at: > > https://cwiki.apache.org/confluence/x/XRvcC < > https://cwiki.apache.org/confluence/x/XRvcC> > > > > Best, > > Zihan Li > >
