Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-12-03 Thread Maxim Fortun
gt; running code on the Kafka brokers themselves. > > I hope this context is helpful. > > Thanks, > Andrew > ________________ > From: Maxim Fortun > Sent: 02 December 2024 22:00 > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-1086: Add ability to specif

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-12-03 Thread Andrew Schofield
e.org Subject: Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser. I could totally use some guidance on how to (who with?) have a discussion about this feature. Are KIPs addressed in the order they are submitted? What is the likelihood of getting this, or a similar fe

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-12-02 Thread Maxim Fortun
I could totally use some guidance on how to (who with?) have a discussion about this feature. Are KIPs addressed in the order they are submitted? What is the likelihood of getting this, or a similar feature into the 4 releases? Again, the purpose of this is to allow to specify a custom produce(m

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-09-03 Thread Maxim Fortun
Based on feedback from Kirk and Colin I added configuring the parser class name via server.properties, added some tests, and updated the docs to reflect this. I find the config file name by re-parsing the command line. If anyone knows a better way of passing KafkaConfig to static initialization,

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-08-29 Thread Maxim Fortun
I totally understand the burden of maintaining features going forward. The backward compatibility breaking behavior you are describing is a normal part of software evolution, just like deprecation. The potential for having to break backward compatibility sometime in the future is hardly a reason

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-08-29 Thread Colin McCabe
Hi Maxim, The issue is not the number of lines of code, but the need to support this indefinitely in the future. For example, if we want to move to a new request type that is different than ProduceRequest, or create a new version of ProduceRequest, we wouldn't be able to do that in the future w

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-08-29 Thread Maxim Fortun
Hi Colin, Thank you for the background. As a new user of Kafka I appreciate being caught up. I tried looking for this in PRs and KIPs but did not find any references. Since my change is tiny, doesn't really change the way the broker works, and only adds an ability to customize the broker without

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-08-29 Thread Colin McCabe
Hi Maxim, Server-side interceptors for messages have been discussed several times, but the KIP never got accepted. People often pushed back on what they saw as the risks and performance problems of allowing user-supplied code in the hot path of accepting records. Your KIP is a bit different be

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-08-29 Thread Maxim Fortun
Oooh! All great questions! Answering inline. > On Aug 29, 2024, at 12:00 PM, Kirk True wrote: > > Hi Maxim, > > Thanks for the KIP! > > This is not my area, but I have some questions/comments: > > 1. CMIIW, but I believe that specifying this custom parser class should be > done via configura

Re: [DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-08-29 Thread Kirk True
Hi Maxim, Thanks for the KIP! This is not my area, but I have some questions/comments: 1. CMIIW, but I believe that specifying this custom parser class should be done via configuration, not system properties or environment variables, right? 2. What happens if the custom parser can’t be loaded?

[DISCUSS] KIP-1086: Add ability to specify a custom produce request parser.

2024-08-29 Thread Maxim Fortun
Hi all, I would like to introduce a minor code change to allow custom produce request parsers. KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=318606528 JIRA: https://issues.apache.org/jira/browse/KAFKA-17348 PR: https://github.com/apache/kafka/pull/16812 There are many po