We have had several requests to add more Connect Single Message Transforms (SMTs) to the project. When SMTs were first introduced with KIP-66 (ref 1) in Jun 2017, the KIP mentioned the following:
> Criteria: SMTs that are shipped with Kafka Connect should be general enough > to apply to many data sources & serialization formats. They should also be > simple enough to not cause any additional library dependency to be introduced. > Beyond those being initially included with this KIP, transformations can be > adopted for inclusion in future with JIRA/ML discussion to weigh the > tradeoffs. In the 4+ years that we've had SMTs in the project, we've only enhanced the framework with KIP-585 (ref 2), and fixed the initial SMTs (including KIP-437, ref 3). We recently have had quite a few requests to add new SMTs; a few samples of these include: * https://issues.apache.org/jira/browse/KAFKA-10299 * https://issues.apache.org/jira/browse/KAFKA-9436 * https://issues.apache.org/jira/browse/KAFKA-9318 * https://issues.apache.org/jira/browse/KAFKA-12443 Adding new or changing existing SMTs to the Apache Kafka project come with requirements. First, AK releases are infrequent and necessarily involve the entire project. Second, adding an SMT is an API change and therefore requires a KIP. Third, all changes in behavior to SMTs included in an prior AK release must be backward compatible, and adding or changing an SMT's configuration requires a KIP. This last one is also challenging if we're limiting ourselves to truly general SMTs, since these are notoriously difficult to get right the first time. All of these aspects mean that it's difficult to add, maintain, and evolve/improve SMTs in AK. And unless a bug fix is critical, we're likely not to create a patch release for AK just to fix a bug in an SMT, simply because of the effort involved. On the other hand, anyone can easily implement their own SMT and deploy them as a Connect plugin, whether that's part of a connector plugin or a separate plugin dedicated for one or SMTs. Interestingly, it's far simpler to implement and maintain custom SMTs outside of AK, especially since those plugins can be released and deployed in any Connect runtime version since at least 0.11.0. And if custom SMTs are maintained in a relatively small project, they can be released often. Finally, KIP-26 (ref 4) specifically rejected maintaining connector implementations in the AK project. So we have precedence for choosing not to accept implementations. Given the above, I wonder if the time has come for us to prefer only maintaining the SMT framework and existing SMTs, and to decline adding new SMTs. Thoughts? Best regards, Randall Hauch (1) https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect (2) https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs (3) https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT (4) https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767
