Jark Wu created FLINK-21191:
-------------------------------
Summary: Support reducing buffer for upsert-kafka sink
Key: FLINK-21191
URL: https://issues.apache.org/jira/browse/FLINK-21191
Project: Flink
Issue Type: New Feature
Components: Connectors / Kafka
Reporter: Jark Wu
Fix For: 1.13.0
Currently, if there is a job agg -> filter -> upsert-kafka, then upsert-kafka
will receive -U and +U for every updates instead of only a +U. This will
produce a lot of tombstone messages in Kafka. It's not just about the
unnecessary data volume in Kafka, but users may processes that trigger side
effects when a tombstone records is ingested from a Kafka topic.
A simple solution would be add a reducing buffer for the upsert-kafka, to
reduce the -U and +U before emitting to the underlying sink. This should be
very similar to the implementation of upsert JDBC sink.
We can even extract the reducing logic out of the JDBC connector and it can be
reused by other connectors.
This should be something like `BufferedUpsertSinkFunction` which has a reducing
buffer and flush to the underlying SinkFunction
once checkpointing or buffer timeout. We can put it in `flink-connector-base`
which can be shared for builtin connectors and custom connectors.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
