Re: Re: [DISCUSS] KIP-905: Broker interceptors

FWIW, this would be very useful for the Wikimedia Foundation's Event Platform. We have some requirements for our event stream producers, and not having to re-implement this logic in multiple programming languages and framew

RE: [DISCUSS] KIP-905: Broker interceptors

Hi David, I wonder if you considered interceptors that work not only on produce request, but on any request type the broker is receiving (apart from maybe internal request types). For example, being able to rewrite topic names in all the request types is essential to create a virtual clusters s

RE: Re: [DISCUSS] KIP-905: Broker interceptors

Hi David and Ahmed, First, thank you David for the KIP. It would be very valuable for multiple use cases. Products like Conduktor Gateway [1] validate the demand and offer many potential use cases [2]. Now, I understand Ahmed's concerns about possible in-band interruptions, the are valid. Howe

Re: [DISCUSS] KIP-905: Broker interceptors

Hello David, Thank you for the proposal - it is an interesting read! I have a few questions about it. 1. Can you take a stance on whether you are proposing the feature just for producers or for consumers as well? If it is just for producers can we remove references to consumers? If it is for bot

Re: [DISCUSS] KIP-905: Broker interceptors

Also, the KIP doesn't describe the client-side experience Is a producer only expected to get the new API error in the response when the interceptor fails unexpectedly ? It looks otherwise that it is expected that records may get skipped or mutated without the producer metadata response noting this

Re: [DISCUSS] KIP-905: Broker interceptors

Hi David thanks for the KIP. Two initial observations from me. I think the Rejected Alternatives section could compare your proposal to the prior art that you rightly mention initially. Also, the Java interface could extend Kafka's own Configurable. This allows an implementation to get hold of t

Re: [DISCUSS] KIP-905: Broker interceptors

Bumping this thread as I'd love to get a bit more feedback on the general approach before proceeding. On Fri, Feb 10, 2023 at 11:41 AM David Mariassy wrote: > Hi Ahmed, > > Thanks for taking a look at the KIP, and for your insightful feedback! > > I don't disagree with the sentiment that in-band

Re: [DISCUSS] KIP-905: Broker interceptors

Hi Ahmed, Thanks for taking a look at the KIP, and for your insightful feedback! I don't disagree with the sentiment that in-band interceptors could be a potential source of bugs in a cluster. Having said that, I don't necessarily think that an in-band interceptor is significantly riskier than a

Re: [DISCUSS] KIP-905: Broker interceptors

Hi David, That's a very interesting KIP and I wanted to share my two cents. I believe there's a lot of value and use cases for the ability to intercept, mutate and filter Kafka's messages, however I'm not sure if trying to achieve that via in-band interceptors is the best approach for this. -

[DISCUSS] KIP-905: Broker interceptors

Hi everyone, I'd like to get a discussion going for KIP-905 , which proposes the addition of broker interceptors to the stack. The KIP contains the motivation, and lists the new public interfaces that this change wo