I like the idea of a select number of SMTs being offered and supported out of the box. The addition of SMTs via this process is nice because it allows for a rich set to be supported out of the box and without the need for extra work to deploy.
Perhaps this is a spot where the community could express the interest of additional SMTs which maybe are available via an open source library and if enough usage occurs there could be a path to fold into the Kafka project at large? Brandon Brown > On Nov 7, 2021, at 1:19 PM, Randall Hauch <rha...@gmail.com> wrote: > > We have had several requests to add more Connect Single Message > Transforms (SMTs) to the project. When SMTs were first introduced with > KIP-66 (ref 1) in Jun 2017, the KIP mentioned the following: > >> Criteria: SMTs that are shipped with Kafka Connect should be general enough >> to apply to many data sources & serialization formats. They should also be >> simple enough to not cause any additional library dependency to be >> introduced. >> Beyond those being initially included with this KIP, transformations can be >> adopted for inclusion in future with JIRA/ML discussion to weigh the >> tradeoffs. > > In the 4+ years that we've had SMTs in the project, we've only > enhanced the framework with KIP-585 (ref 2), and fixed the initial > SMTs (including KIP-437, ref 3). We recently have had quite a few > requests to add new SMTs; a few samples of these include: > * https://issues.apache.org/jira/browse/KAFKA-10299 > * https://issues.apache.org/jira/browse/KAFKA-9436 > * https://issues.apache.org/jira/browse/KAFKA-9318 > * https://issues.apache.org/jira/browse/KAFKA-12443 > > Adding new or changing existing SMTs to the Apache Kafka project come > with requirements. First, AK releases are infrequent and necessarily > involve the entire project. Second, adding an SMT is an API change and > therefore requires a KIP. Third, all changes in behavior to SMTs > included in an prior AK release must be backward compatible, and > adding or changing an SMT's configuration requires a KIP. This last > one is also challenging if we're limiting ourselves to truly general > SMTs, since these are notoriously difficult to get right the first > time. All of these aspects mean that it's difficult to add, maintain, > and evolve/improve SMTs in AK. And unless a bug fix is critical, we're > likely not to create a patch release for AK just to fix a bug in an > SMT, simply because of the effort involved. > > On the other hand, anyone can easily implement their own SMT and > deploy them as a Connect plugin, whether that's part of a connector > plugin or a separate plugin dedicated for one or SMTs. Interestingly, > it's far simpler to implement and maintain custom SMTs outside of AK, > especially since those plugins can be released and deployed in any > Connect runtime version since at least 0.11.0. And if custom SMTs are > maintained in a relatively small project, they can be released often. > > Finally, KIP-26 (ref 4) specifically rejected maintaining connector > implementations in the AK project. So we have precedence for choosing > not to accept implementations. > > Given the above, I wonder if the time has come for us to prefer only > maintaining the SMT framework and existing SMTs, and to decline adding > new SMTs. > > Thoughts? > > Best regards, > > Randall Hauch > > (1) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect > (2) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs > (3) > https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT > (4) https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767
