Hi all,
I think restricting the set of out-of-the-box SMTs that we provide with
Connect is reasonable. I do think Joshua raises a valuable point, though.
At the risk of reiterating his ideas, we can gain a few things from
improving the existing SMTs provided with Connect: first, we can establish
precedents for how SMTs are configured and implemented in more complex
scenarios (such as handling explicitly-specified nested fields or
traversing an entire key/value recursively), which can save time for both
developers and users if we do a good enough job for others to start
following the examples we set. Second, we decrease the likelihood that
someone forks, e.g., the InsertField SMT just to add their own small tweak
on top, which both adds unnecessary work for that developer and complicates
the experience for users of Connect ("which SMT do I use now?").
Additionally, I like Gunnar and Brandon's suggestion of a way to discover
SMTs. There's precedent for this with the "Kafka Connector Hub" link on the
https://cwiki.apache.org/confluence/display/KAFKA/ page, which currently
leads to a page on Confluent's website containing a fairly large list of
connectors from a variety of sources (
https://www.confluent.io/product/connectors/). In practice I'm not sure how
many new Kafka users end up visiting the wiki as their first stop, though.
Perhaps we could add a section to the docs page at
https://kafka.apache.org/documentation.html, for connectors,
transformations, and maybe even other pluggable components (converters,
config providers, etc.)?
Cheers,
Chris
On Sun, Nov 21, 2021 at 12:05 PM Joshua Grisham
wrote:
> Hi all,
>
> From my perspective I think that the type of transformations which are
> already covered by the existing SMTs is quite good (but anyone else please
> say if you feel like you are missing something that feels "standard"), but
> the biggest issue is the limitations that many of them have which makes
> their usage extremely limited when trying to use them in a real production
> scenario.
>
> In my mind, the single biggest gap is the inability to handle nested fields
> or anything more than records that essentially look like simple key-value
> pairs. (However one exception being if you chain the flatten transform
> first then you can apply others on the flattened result, but this is
> assuming that the flatten transform can actually handle the message first!
> If you have nested arrays then you are toast ;) And wait, maybe you didn't
> actually want to flatten anyway?).
>
> I am not sure the best way to approach this (e.g. allow for some kind of
> path notation so users can address nested fields directly vs allow for
> recursion to match a field name at no matter what level, or both, or
> something else?) but I would say that some kind of standardized approach
> that was implemented in all of the SMTs (where it makes sense) would
> certainly be best! (at least, from a user perspective that the
> configuration to address nested fields is consistent across each transform
> that allows it). I did this one way in a proposed change for KIP-683 but
> this is only one of the possible ways (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-683%3A+Add+recursive+support+to+Connect+Cast+and+ReplaceField+transforms%2C+and+support+for+casting+complex+types+to+either+a+native+or+JSON+string
> )
>
> Past that, there are a few tweaks or enhancements which could be made to
> some of the existing SMTs which would help prevent them from blocking or
> failing for most general scenarios (for example some of the changes I had
> proposed in the past but haven't since had the time to follow up on them
> fit in this category I think), for example the ability to "cast" a more
> complicated structure (such as an array) as a string (Connect API or JSON)
> so the record can then be flattened and be inserted into a database table
> or something similar will open up a lot of what is IMO currently roadblocks
> that users might often hit in Sink scenarios.
>
> Then there are some small tweaks which maybe can be made for specific
> cases, some of which Randall already mentioned, such as:
>
> * The Filter implementation is very limited to use mostly due to lack of
> some "standard-feeling" predicates (field value filtering is very often
> what I think people are looking for) so often the Confluent or other one is
> used instead.
> * A bit more can be done with InsertField IMO (e.g. giving a wallclock
> timestamp instead of the record's produced timestamp is one example that
> often seems to pop up).
> * Some standardized way to "move" one field to another place e.g. to move
> it out of or into a nested record.
> * Limitations on only processing one field per transformation, e.g. with
> the TimestampConverter like I had proposed with KIP-682 (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-682%3A+Connect+TimestampConverter+support+for+multiple+fields+and+multiple+input+formats
> )
> are just a little annoying
