[
https://issues.apache.org/jira/browse/FLINK-33783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798873#comment-17798873
]
Bartosz Mikulski commented on FLINK-33783:
------------------------------------------
Hi [~martijnvisser]!
Sorry for the delayed response. I agree that this topic should be a FLIP at
some point! Can I create such FLIP now or does it require a bigger discussion
first?
I created this issue first as a stopgap solution. I already implemented the
feature in a fork, it can be found here:
[https://github.com/deep-bi/flink-connector-kafka/releases/tag/deep-v3.0.2].
Maybe opening a pull request would be a great first step for starting a wider
discussion about DLQ (Dead Letter Queue) handling in SQL. What do you think?
Regarding your side note, I agree with you. There should be little to no chance
of having unparsable events. However, our client use case is a bit different -
they want to manage the schemas themselves and disable the auto-register
feature (see: https://issues.apache.org/jira/browse/FLINK-33045). In this
scenario, they want to control the schema registry outside of the Flink. They
also use other things in the pipeline, like Kafka Streams, etc. If some events
end up in the topic that are not registered they should be ignored. Again, this
is still more of a hypothetical scenario, but it still may happen at some point.
> Add options to ignore parsing error in Kafka SQL Connector
> ----------------------------------------------------------
>
> Key: FLINK-33783
> URL: https://issues.apache.org/jira/browse/FLINK-33783
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Table SQL / API
> Reporter: Bartosz Mikulski
> Priority: Major
>
> h1. Current state
> If an unparsable event enters a Flink Kafka Source in SQL then the whole
> application restarts. For JSON format there is a property that allows to
> ignore unparsable events. Other formats, like `confluent-avro` etc don't
> support that.
> h1. Desired state
> We would like to ignore the parsing exception in Kafka Source in SQL
> regardless of the format used. Additionally, a new metric should be
> introduced that returns a count of unparsable events seen so far.
> In the future there should be a Dead Letter Queue handling in SQL Sources
> similar to Kafka Streams:
> [https://www.confluent.io/blog/kafka-connect-deep-dive-error-handling-dead-letter-queues/].
> For now, the universal ignore with metric would be enough.
> h1. Implementation
> We already have an implementation for this case in the Flink Kafka Connector
> and we would like to open a pull request for it. However, we created the
> issue first as per this recommendation
> [https://flink.apache.org/how-to-contribute/contribute-code/]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
