Diego Erdody created KAFKA-13431:
------------------------------------
Summary: Sink Connectors: Support topic-mutating SMTs for async
connectors (preCommit users)
Key: KAFKA-13431
URL: https://issues.apache.org/jira/browse/KAFKA-13431
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Reporter: Diego Erdody
Assignee: Diego Erdody
There's currently an incompatibility between Sink connectors overriding the
{{SinkTask.preCommit}} method (for asynchronous processing) and SMTs that
mutate the topic field.
The problem was present since the {{preCommit}} method inception and is rooted
in a mismatch between the topic/partition that is passed to {{open/preCommit}}
(the original topic and partition before applying any transformations) and the
topic partition that is present in the SinkRecord that the {{SinkTask.put}}
method receives (after transformations are applied). Since that's all the
information the connector has to implement any kind of internal offset
tracking, the topic/partitions it can return in preCommit will correspond to
the transformed topic, when the framework actually expects it to be the
original topic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
