Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Tom, I added Configuration Properties sections that document the properties and a couple of examples. On Tue, Jun 28, 2022 at 10:57 AM Tom Bentley wrote: > Hi Omnia, > > Please could you add to the KIP the documentation that forward.admin.class > config will have, and include that the signatu

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia, Please could you add to the KIP the documentation that forward.admin.class config will have, and include that the signature of the class named by forward.admin.class must have a constructor that takes a `Map`? Many thanks, Tom On Wed, 22 Jun 2022 at 11:30, wrote: > Hi Tom > My initi

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Tom My initial thought was that the constructor with config and delegator would be for testing while the one with config is the one that will be used by MM2. I can removed one of them and keep only one. Sent from my iPhone > On 22 Jun 2022, at 08:15, Tom Bentley wrote: > > Hi Omnia, > >

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia, Thanks for those answers. I'm a bit confused by the changes you've made for 1 though. It's MM2 that's going to instantiate the class named in forwarding.admin.class, so it's MM2 that needs to know the constructor signature. The easiest way of doing this is to specify the signature as par

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Tom > 1. I assume there's an undocumented requirement in the KIP that whatever > class is named for forwarding.admin.class it has a public single argument > constructor that takes an Admin instance? > you are right, I updated the KIP to reflect the missing one. 3. What if the implementation nee

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia, Thanks for the KIP! I'm sorry for the delay in this response. I have a few questions: 1. I assume there's an undocumented requirement in the KIP that whatever class is named for forwarding.admin.class it has a public single argument constructor that takes an Admin instance? 2. If 1 is c

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Chris, sorry for the late reply. ..Hi, > 1. I might be missing something, but can you give a concrete Java example > of how the proposed ForwardingAdmin class is more convenient than > subclassing the KafkaAdminClient class? AFAICT the two would be virtually > identical. I might be misundersta

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia, 1. I might be missing something, but can you give a concrete Java example of how the proposed ForwardingAdmin class is more convenient than subclassing the KafkaAdminClient class? AFAICT the two would be virtually identical. I might be misunderstanding exactly how that class will be used

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Chris, Thanks for having the time to look into this. 1. Is the introduction of the new "ForwardingAdmin" class necessary, or can > the same behavior can be achieved by subclassing the existing > KafkaAdminClient class? forwarding decorators give more flexibility than the inheritance, in this c

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia, Thank you for your patience with this KIP! I have a few quick thoughts: 1. Is the introduction of the new "ForwardingAdmin" class necessary, or can the same behavior can be achieved by subclassing the existing KafkaAdminClient class? 2. Would it be just as accurate to name the new Mirr

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi everyone, If there's no major concern anymore, I'll start the voting process. On Fri, May 20, 2022 at 5:58 PM Omnia Ibrahim wrote: > Hi Colin, > > >Thanks for the clarification. I agree it's reasonable for people to want > to use their own implementations of Admin. And we could have a config

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Colin, >Thanks for the clarification. I agree it's reasonable for people to want to use their own implementations of Admin. And we could have a config for this, so that it becomes pluggable (possibly in other places than MirrorMaker, although we don't have to do that in this KIP). > Allowing pe

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

On Mon, May 16, 2022, at 10:24, Omnia Ibrahim wrote: > Hi Colin, > > Thanks for your reply. > > This KIP doesn’t aim to solve any security concerns, but rather a conflict > of responsibilities within any Kafka ecosystem that includes MM2 and any > resource management solution. I’m not sure that was

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Colin, Thanks for your reply. This KIP doesn’t aim to solve any security concerns, but rather a conflict of responsibilities within any Kafka ecosystem that includes MM2 and any resource management solution. I’m not sure that was clear, so I’m concerned about the motivation for your suggestion

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

On Wed, May 11, 2022, at 15:07, Omnia Ibrahim wrote: > Hi Colin, > I don't mind the idea of MM2 users implementing the AdminClient interface. > However, there're two disadvantages to this. > >1. Having around 70 methods definitions to have "NotImplemented" is one >downside, and keep up with

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

I updated the KIP to reflect the options we have been discussing since Oct 2021 for people who didn't read the discussion thread. On Wed, May 11, 2022 at 11:07 PM Omnia Ibrahim wrote: > Hi Colin, > I don't mind the idea of MM2 users implementing the AdminClient interface. > However, there're two

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Colin, I don't mind the idea of MM2 users implementing the AdminClient interface. However, there're two disadvantages to this. 1. Having around 70 methods definitions to have "NotImplemented" is one downside, and keep up with these if the AdminClient interface changes. 2. It makes it h

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia Ibrahim, I'm sorry, but I am -1 on adding competing Admin interfaces. This would create confusion and a heavier maintenance burden for the project. Since the org.apache.kafka.clients.admin.Admin interface is a Java interface, any third-party software that wants to insert its own implem

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi, I gave the KIP another look after talking to some people at the Kafka Summit in London. And I would like to clear up the motivation of this KIP. At the moment, MM2 has some opinionated decisions that are creating issues for teams that use IaC, federated solutions or have a capacity/budget pla

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Mickael. Thanks for the feedback. I address some of your points below. *> This seems to address a relatively advanced and specific use case* The main point of the KIP is that MM2 is making a massive assumption that it has the right to alter/create resources. This assumption isn't valid in the w

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi Omnia, Thanks for the KIP, sorry for taking so long to comment. I've only had time to take a quick look so far. This seems to address a relatively advanced and specific use case. My initial concern is this may make it hard to evolve MirrorMaker as we'll likely need to update this new interface

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hi, If there are no more concerns regarding the proposal can I get some votes on the KIP here https://lists.apache.org/thread/950lpxjb5kbjm8qdszlpxm9h4j4sfyjx Thanks On Wed, Oct 27, 2021 at 3:54 PM Ryanne Dolan wrote: > Well I'm

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Well I'm convinced! Thanks for looking into it. Ryanne On Wed, Oct 27, 2021, 8:49 AM Omnia Ibrahim wrote: > I checked the difference between the number of methods in the Admin > interface and the number of methods MM2 invokes from Admin, and this diff > is enormous (it's more than 70 methods).

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

I checked the difference between the number of methods in the Admin interface and the number of methods MM2 invokes from Admin, and this diff is enormous (it's more than 70 methods). As far as I can see, the following methods MM2 depends on (in MirrorSourceConnector, MirrorMaker, MirrorCheckpointTa

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

I like the idea of failing-fast whenever a custom impl is provided, but I suppose that that could be done for Admin as well. I agree your proposal is more ergonomic, but maybe it's okay to have a little friction in such fringe use-cases. Ryanne On Tue, Oct 26, 2021, 6:23 AM Omnia Ibrahim wrote:

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hey Ryanne, Thanks fo the quick feedback. Using the Admin interface would make everything easier, as MM2 will need only to configure the classpath for the new implementation and use it instead of AdminClient. However, I have two concerns 1. The Admin interface is enormous, and the MM2 users will ne

Re: [DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Thanks Omnia, neat idea. I wonder if we could use the existing Admin interface instead of defining a new one? Ryanne On Mon, Oct 25, 2021, 12:54 PM Omnia Ibrahim wrote: > Hey everyone, > Please take a look at KIP-787 > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-787%3A+MM2+Interface

[DISCUSS] KIP-787 - MM2 Interface to manage Kafka resources

Hey everyone, Please take a look at KIP-787 https://cwiki.apache.org/confluence/display/KAFKA/KIP-787%3A+MM2+Interface+to+manage+Kafka+resources Thanks for the feedback and support Omnia