Hello David, Thank you for the proposal - it is an interesting read!
I have a few questions about it. 1. Can you take a stance on whether you are proposing the feature just for producers or for consumers as well? If it is just for producers can we remove references to consumers? If it is for both can you explicitly call out the properties used to configure the consumers? 2. What about extending this to altering configurations? One shortcoming of Kafka today is that a range of values of one configuration affect the range of values another configuration can have, but the validation framework within Kafka does not have the capability to make such checks. 3. Does it not make sense for the pattern which determines whether an interceptor is to be applied or not to be a configuration? Otherwise if there is a problem with the pattern I have to carry out a whole new deployment since I need to change the code. In the same line of reasoning, will there be a way I can query the cluster to understand what interceptors are currently running and what patterns they are using? Otherwise how would I know what is the cluster's current configuration? 4. Will there be any new metrics emitted by said interceptors (i.e. number of records dropped, number of records processed per unit time)? If there aren't how will I be able to determine the performance of my interceptors? If I have multiple interceptors how will I be able to determine which one is the bottleneck? 5. Will records pass through interceptors in the same order as the interceptors are specified in the list or will there be another way to specify the ordering? 6. What happens if you have a pipeline of interceptors and some of them are the same, how will you handle loading the interceptors then? For example, I can imagine someone saying filter everything above a value X, do some more complex operation, filter everything above a value X, do another more complex operation, filter everything above a value X etc. Let me know your thoughts! Best, Christo On Thu, 9 Feb 2023 at 19:28, David Mariassy <david.maria...@gmail.com> wrote: > Hi everyone, > > I'd like to get a discussion going for KIP-905 > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-905%3A+Broker+interceptors > >, > which proposes the addition of broker interceptors to the stack. > > The KIP contains the motivation, and lists the new public interfaces that > this change would entail. Since my company had its quarterly hack days this > week, I also took the liberty to throw together a first prototype of the > proposed new feature here: https://github.com/apache/kafka/pull/13224. > > Looking forward to the group's feedback! > > Thanks, > David >
