We have had several requests to add more Connect Single Message
Transforms (SMTs) to the project. When SMTs were first introduced with
KIP-66 (ref 1) in Jun 2017, the KIP mentioned the following:

> Criteria: SMTs that are shipped with Kafka Connect should be general enough 
> to apply to many data sources & serialization formats. They should also be 
> simple enough to not cause any additional library dependency to be introduced.
> Beyond those being initially included with this KIP, transformations can be 
> adopted for inclusion in future with JIRA/ML discussion to weigh the 
> tradeoffs.

In the 4+ years that we've had SMTs in the project, we've only
enhanced the framework with KIP-585 (ref 2), and fixed the initial
SMTs (including KIP-437, ref 3). We recently have had quite a few
requests to add new SMTs; a few samples of these include:
* https://issues.apache.org/jira/browse/KAFKA-10299
* https://issues.apache.org/jira/browse/KAFKA-9436
* https://issues.apache.org/jira/browse/KAFKA-9318
* https://issues.apache.org/jira/browse/KAFKA-12443

Adding new or changing existing SMTs to the Apache Kafka project come
with requirements. First, AK releases are infrequent and necessarily
involve the entire project. Second, adding an SMT is an API change and
therefore requires a KIP. Third, all changes in behavior to SMTs
included in an prior AK release must be backward compatible, and
adding or changing an SMT's configuration requires a KIP. This last
one is also challenging if we're limiting ourselves to truly general
SMTs, since these are notoriously difficult to get right the first
time. All of these aspects mean that it's difficult to add, maintain,
and evolve/improve SMTs in AK. And unless a bug fix is critical, we're
likely not to create a patch release for AK just to fix a bug in an
SMT, simply because of the effort involved.

On the other hand, anyone can easily implement their own SMT and
deploy them as a Connect plugin, whether that's part of a connector
plugin or a separate plugin dedicated for one or SMTs. Interestingly,
it's far simpler to implement and maintain custom SMTs outside of AK,
especially since those plugins can be released and deployed in any
Connect runtime version since at least 0.11.0. And if custom SMTs are
maintained in a relatively small project, they can be released often.

Finally, KIP-26 (ref 4) specifically rejected maintaining connector
implementations in the AK project. So we have precedence for choosing
not to accept implementations.

Given the above, I wonder if the time has come for us to prefer only
maintaining the SMT framework and existing SMTs, and to decline adding
new SMTs.

Thoughts?

Best regards,

Randall Hauch

(1) 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-66%3A+Single+Message+Transforms+for+Kafka+Connect
(2) 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Filter+and+Conditional+SMTs
(3) 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-437%3A+Custom+replacement+for+MaskField+SMT
(4) https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767

Reply via email to