Haohui Mai created FLINK-5583:
---------------------------------
Summary: Support flexible error handling in the Kafka consumer
Key: FLINK-5583
URL: https://issues.apache.org/jira/browse/FLINK-5583
Project: Flink
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
We found that it is valuable to allow the applications to handle errors and
exceptions in the Kafka consumer in order to build a robust application in
production.
The context is the following:
(1) We have schematized, Avro records flowing through Kafka.
(2) The decoder implements the DeserializationSchema to decode the records.
(3) Occasionally there are corrupted records (e.g., schema issues). The
streaming pipeline might want to bail out (which is the current behavior) or to
skip the corrupted records depending on the applications.
Two options are available:
(1) Have a variant of DeserializationSchema to return a FlatMap like structure
as suggested in FLINK-3679.
(2) Allow the applications to catch and handle the exception by exposing some
APIs that are similar to the {{ExceptionProxy}}.
Thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
