Hi Konstantine,
Thanks very much for your input. It's helpful to have a steer from a
committer on the syntax point especially. I've made all those changes and
added a note about compatibility.
Cheers,
Tom
On Thu, May 14, 2020 at 6:00 AM Konstantine Karantasis <
konstant...@confluent.io> wrote:
Hi Tom.
Thanks for the KIP. I like how the proposal has ended up to be and I think
it describes a practical approach.
I have to say that, for a moment, earlier in the discussion I thought we
were leaning a bit towards an unconventional mini assembly language based
on java properties.
The reliance
Hi Andrew,
That works nicely enough for the proposal where the predicate is configured
directly on the transformation. But I thought there was more consensus
around the proposal to have the transformation configuration refer to a
predicate indirectly, defined with the ?predicates key. I guess an e
Hi,
I have implemented some of this and configured some predicates and I prefer
this slight modification of the ? syntax:
transforms: t2
transforms.t2?type:
org.apache.kafka.connect.transforms.predicates.TopicNameMatch
transforms.t2?negate: true
transforms.t2?pattern: my-prefix-.*
transforms.t2.
Hi Tom,
I think that either proposal is sufficiently readable and they both have
elements of
cryptic syntax. I remember talking to an engineering director once who said he
didn't hire
"curly braces people" any more. This is an interface for curly braces people
whichever
way we go here.
I slight
Hi,
Again, reusing the existing numbering...
3) I had exactly the same thought. The reason I didn't already rename it
was because :
* Filter is not exactly wrong (by default it just filters out all the
messages)
* Filter is a useful hint about how it's intended to be used.
But I don't have a st
Hi Tom,
Referring back to Chris's points.
1) Exception handling - yes, agreed. This KIP can just say the exception
handling for
predicates is the same as transformations.
2) Worker plugins - agree too.
3) Filter SMT - If the Filter SMT is just becoming a transformation that drops
any records t
Hi Chris,
Thanks for the feedback. On points 1 and 2 I agree.
3. This is a great point. I'm happy to add this to the KIP too, but note it
removes the need to add validate(Map) to both
Transformation and Predicate. That doesn't mean we shouldn't still do it,
just that the change is not motivated b
Hi all,
This has certainly taken quite a turn! I wish we could get this done
without adding another pluggable interface but if the benefit is that now
any SMT--new or pre-existing--can be conditionally applied, it seems like
it might be worth it.
As far as the proposed design goes, some thoughts:
Hi Tom,
Looks good to me.
Thanks,
Andrew
On 01/05/2020, 16:24, "Tom Bentley" wrote:
Hi,
I've updated the KIP to reflect recent discussions. Please note the
following:
1. I've described a HasHeaderKey predicate rather than HeaderKeyMatches
because at the moment Headers doe
Hi,
I've updated the KIP to reflect recent discussions. Please note the
following:
1. I've described a HasHeaderKey predicate rather than HeaderKeyMatches
because at the moment Headers doesn't expose the whole set of headers, only
those with a specific name. Exposing this would significantly incr
Hi Tom,
I think we should go for a consistent naming convention for the predicates,
maybe so they
read nicely if you say "IF" first. Here's a suggestion for predicates that the
KIP could introduce:
- TopicNameMatches
- HeaderKeyMatches
- RecordIsTombstone
On naming, I also suggest using org.a
Hi Andrew and Chris,
Firstly, thanks for the input.
To me, the ?type syntax has some pros and cons. On the pros side:
* it's pretty succinct
* it's flexible enough for the use cases we've identified so far.
On the cons side:
* it's a bit cryptic; I don't think people who didn't know it already w
Hi,
I'm suggesting that RecordPredicate be another pluggable interface that users
could conceivably implement on their own, and that the KIP introduces the
most likely ones as supplied implementations, much as there are SMT
implementations such as HoistField. I don't really see users implementing
t
Hi Andrew,
I know a DSL seems like overkill, and I'm not attached to it by any means,
but I do think it serves a vital purpose in that it allows people who don't
know how or have the time to write Java code to act on data being processed
by connectors.
Are you proposing that the "RecordPredicate"
I wonder whether we're getting a bit overcomplicated with this. I think all
that's required here is to add an optional guard predicate for a Transformation.
The predicate cannot end the Transformation chain, but it can allow or prevent
a particular
Transformation from running.
How about this as s
Hi Tom,
With regards to the potential Transformation::validate method, I don't
quite follow your objections. The AbstractHerder class, ConnectorConfig
class, and embedding of ConfigDefs that happens with the two is all
internal logic and we're allowed to modify it however we want, as long as
it do
Since no one objected I've updated the KIP with the revised way to
configure this transformation.
On Mon, Apr 6, 2020 at 2:52 PM Tom Bentley wrote:
> To come back about a point Chris made:
>
> 1. Instead of the new "ConfigDef config(Map props)" method,
>> what would you think about adopting a si
To come back about a point Chris made:
1. Instead of the new "ConfigDef config(Map props)" method,
> what would you think about adopting a similar approach as the framework
> uses with connectors, by adding a "Config validate(Map
> props)" method that can perform custom validation outside of what
Hi,
Hi all,
Thanks for the discussion so far.
It seems a bit weird to me that when configuring the Conditional SMT with a
DSL you would use a concise, intuitive DSL for expressing the condition,
but not for the transforms that it's guarding. It also seems natural, if
you support this for conditi
Hi all,
Thanks a lot for this initiative, Tom!
To shed some light, the use case where this first came up, were issues
we saw with SMTs being applied to the different topics produced by the
Debezium change data capture connectors. There are different kinds of
topics (for change data, schema histor
Hi Tom, hi Chris,
in principle an expression language sounds good, as it would allow us to
cover pretty much any use-case that we could possibly come up with. And
bearing in mind that I was the one who originally started talking about
more complex use-cases, I should be happy about that :)
That b
Hi Chris and Sönke,
Using the numbering from Chris's email...
1. That's a good point, I'll see what is needed to make that work.
2. I'm happy enough to add support for "and" and "or" as part of this KIP
if people can see a need for it.
In a similar vein, I was wondering about whether it would b
Hi Tom,
thanks for your detailed explanation!
I may have mis-phrased my mail a little bit regarding the definition of
conditions and caused you extra effort with that, sorry! I understood the
not condition and its relationship to the has-header condition, I was just
wondering whether the "" in
"tr
Hi Tom,
This looks great and I'd love to see the out-of-the-box SMTs become even
more powerful with the improvements you've proposed! Got a few remarks and
would be interested in your thoughts:
1. Instead of the new "ConfigDef config(Map props)" method,
what would you think about adopting a simil
Hi Sönke,
Thanks for taking a look.
Let me answer in reverse order, since I think it might make more sense that
way...
Also, while writing that, I noticed that you have a version with and
> without "name" for your transformation in the KIP:
>
> transforms.conditionalExtract.condition.hasMyHeader
Hi Tom,
sounds useful to me, thanks for the KIP!
The only thought that I had while reading was that this will probably raise
questions about more involved conditions fairly quickly. For example the
"has-header" will cause an appetite for conditions like
"this-header-has-that-value".
This would nec
Does anyone have any comments, feedback or thoughts about this?
Thanks,
Tom
On Tue, Mar 24, 2020 at 11:56 AM Tom Bentley wrote:
> Hi,
>
> I've opened KIP-585 which is intended to provide a mechanism to
> conditionally apply SMTs in Kafka Connect. I'd be grateful for any
> feedback:
> https://c
Hi,
I've opened KIP-585 which is intended to provide a mechanism to
conditionally apply SMTs in Kafka Connect. I'd be grateful for any
feedback:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Conditional+SMT
Many thanks,
Tom
29 matches
Mail list logo
