JNSimba commented on PR #68: URL: https://github.com/apache/doris-kafka-connector/pull/68#issuecomment-2826601626
> > Thank you for your contribution. -> Because in `org.apache.kafka.connect.sink.SinkTask#preCommit`, the topic corresponds to the original topic before any transformations are applied. If the connector is restarted, the data will be consumed repeatedly. > > Does it mean that the topic in `SinkTask#preCommit` is the original topic name? If so, it seems that **"transforms": "AddPrefix"** will never take effect, because the new topic can never be obtained? What are the common application scenarios of this? Could you please introduce them? Or is there a relevant configuration that allows the topic in precommit to be the transformed topic name? > > @JNSimba Yes, in the code comments of `org.apache.kafka.connect.sink.SinkTask#preCommit`, it is noted that the offset parameter is the original Kafka topic partition and offset, before any transforms have been applied, and no configuration is provided to allow modification of this behavior. However, transforms like `RegexRouter` still take effect because in the `put` method, the received record is the result of applying the transforms. If the topic name is modified, the offset obtained through the original topic in `preCommit` is incorrect. As for the application scenario of renaming topics, many sink connectors use the topic name as the default table name written. If users do not customize mapping and dynamically route tables by renaming topics, problems may arise. Of course, Doris Kafka connector supports dynamic routing through `record.tablename.field`, but I think the scenario of renaming a topic should throw an exception to make the user aware. @chuang-wang-pre Thanks for your explanation, I left some comments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@doris.apache.org For additional commands, e-mail: dev-h...@doris.apache.org
