Re: MM2 taking into consideration automatic topic creation property from original cluster

2021-06-14 Thread David Beech
Hi Omnia Maybe - I'll look forward to reading the KIP when it's submitted. When I researched the feature request from our client I found KIP-158 which added a bunch of topic auto-creation controls to the Kafka Connect framework in version 2.6. It seemed to me like a convenient solution would be t

Re: MM2 taking into consideration automatic topic creation property from original cluster

2021-06-14 Thread Omnia Ibrahim
Hi, I am in the middle of writing a new KIP to introduced a new interface for topic updates/creation/description that will allow MM2 to either uses the default behaviour which uses Kafka AdminClient to create/update topics or to use a customised one that integrates with the user's ecosystem where t

Re: MM2 taking into consideration automatic topic creation property from original cluster

2021-06-14 Thread Igor Soarez
Maybe it would be nice if it was possible to hook into or extend Admin client interactions, to allow for custom logic supporting use cases such as this. Scenarios where topic/resource management is centralized are probably not uncommon. -- Igor On Sat, Jun 12, 2021, at 9:42 AM, Matthew de Detri

Re: MM2 taking into consideration automatic topic creation property from original cluster

2021-06-12 Thread Matthew de Detrich
Thanks Ryanne and Mickael I already suspected that something like this (i.e. current behaivior being a deliberate design decision) was the case and I just wanted to confirm my suspicions. I will relay this internally On Fri, 11 Jun 2021, 18:55 Mickael Maison, wrote: > Hi Matthew, > > If an adm

Re: MM2 taking into consideration automatic topic creation property from original cluster

2021-06-11 Thread Mickael Maison
Hi Matthew, If an administrator want to control topic creation, they should use ACLs to prevents users creating topic. Relying on all applications to not create topics is unlikely to succeed. Each refresh.topics.interval.seconds, MM2 checks if it needs to create topics/partitions by comparing bot

Re: MM2 taking into consideration automatic topic creation property from original cluster

2021-06-11 Thread Ryanne Dolan
Matthew, I wonder what the expected behavior would be when topic-creation is disabled and MM is asked to replicate a topic that doesn't exist on the target cluster? Maybe the task should fail at that point, or maybe it should replicate whatever it can? I think the current behavior is reasonable, e

MM2 taking into consideration automatic topic creation property from original cluster

2021-06-11 Thread Matthew de Detrich
Hello everyone, We have an interesting feature request from a client regarding having the property of automatic topic creation to be reflected in a MM2. Specifically the current behaviour where if you have automatic topic creation set to false for the original Kafla cluster, MM2 configuration igno