solve this is under-discussion in
> [KIP-690|https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
t; >>> > --
> > > >> > >>> > Igor
> > > >> > >>> >
> > > >> > >>> >
> > > >> > >>> > On Fri, Apr 30, 2021, at 3:15 PM, Ryanne Dolan wrote:
> > > >>
M, Ryanne Dolan wrote:
> > >> > >>> > > +1 (non-binding), thanks!
> > >> > >>> > >
> > >> > >>> > > On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim <
> > >> > o.g.h.
an wrote:
> >> > >>> > > +1 (non-binding), thanks!
> >> > >>> > >
> >> > >>> > > On Thu, Jan 21, 2021, 4:31 AM Omnia Ibrahim <
> >> > o.g.h.ibra...@gmail.com>
> >> > >>> wrote:
> >
31 AM Omnia Ibrahim <
>> > o.g.h.ibra...@gmail.com>
>> > >>> wrote:
>> > >>> > >
>> > >>> > >> Hi
>> > >>> > >> Can I get a vote on this, please?
>> > >>> > >>
&g
; o.g.h.ibra...@gmail.com>
> > >>> wrote:
> > >>> > >
> > >>> > >> Hi
> > >>> > >> Can I get a vote on this, please?
> > >>> > >>
> > >>> > >> Best
> >
>>> > >
> >>> > >> Hi
> >>> > >> Can I get a vote on this, please?
> >>> > >>
> >>> > >> Best
> >>> > >> Omnia
> >>> > >>
> >>> > >
Thanks for the KIP!
+1 (non-binding) from my side!
On Tue, Jul 27, 2021 at 2:22 PM lobo xu wrote:
> +1，I think that's a good idea. When we use MirrorMaker2, we do not want
> subject name conversion to occur. Hopefully there is a switch that can be
> configured.
>
> > 在 2021年7月27日，06:38，Omnia Ibr
+1，I think that's a good idea. When we use MirrorMaker2, we do not want subject
name conversion to occur. Hopefully there is a switch that can be configured.
> 在 2021年7月27日，06:38，Omnia Ibrahim 写道：
>
> Bumping up this voting thread.
e?
>>> > >>
>>> > >> Best
>>> > >> Omnia
>>> > >>
>>> > >> On Tue, Dec 15, 2020 at 12:16 PM Omnia Ibrahim <
>>> o.g.h.ibra...@gmail.com>
>>> > >> wrote:
>>> > >&
:
>> > >>
>> > >>> If anyone interested in reading the discussions you can find it here
>> > >>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
>> > >>>
>> > >>> On Tue, Dec 8, 2020 a
om>
> > >> wrote:
> > >>
> > >>> If anyone interested in reading the discussions you can find it here
> > >>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
> > >>>
> > >>> On Tue, Dec 8, 2020
y that if
> this team
> >> >>> has a naming topic's rule, this same rule will be applied to both
> >> >>> replicated and some of the internal topics of MM2. If we added
> config
> >> >>> like connect to each internal to
>
> >>> If anyone interested in reading the discussions you can find it here
> >>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
> >>>
> >>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim
> >>> wrote:
> >>>
> &
rule, 1 to include customised
>> >>> replication policy for naming replicated topics + extra 3 configs for the
>> >>> internals to match the same rule.
>> >>>
>> >>> 2. The replication policy plugin already implemented in the MM2, and
>>
nfig customers need to add.
> >>>
> >>> Omnia
> >>>
> >>>
> >>> On Thu, May 13, 2021 at 9:30 PM Gwen Shapira >
> >>> wrote:
> >>>
> >>>> Hey, sorry for arriving late, but I have a question about the re
e are looking at 3 new configs (one for
>>>> each topic). And this rejected alternative is basically identical to
>>>> what
>>>> Connect already does (you can choose names for internal topics using
>>>> configs). How is it too complex? As a user, configuring 3 fie
>>>
>>> > Hi Ryanne, Can you vote for it here
>>> > https://www.mail-archive.com/dev@kafka.apache.org/msg113575.html as
>>> well,
>>> > please?
>>> >
>>> > On Fri, Apr 30, 2021 at 12:44 AM Ryanne Dolan
>>> > wro
>> > >> I updated the KIP
>> > >>
>> > >> On Thu, Apr 29, 2021 at 4:43 PM Omnia Ibrahim <
>> o.g.h.ibra...@gmail.com>
>> > >> wrote:
>> > >>
>> > >>> Sure, this would make it easier, we can make these funct
>> customisation using `replication.policy.separator` and use the
> > separator in
> > >>> the DefaultReplicationPolicy
> > >>>
> > >>> On Wed, Apr 28, 2021 at 1:31 AM Ryanne Dolan
> > >>> wrote:
> > >>>
> >
here
> >>>>>>>> https://www.mail-archive.com/dev@kafka.apache.org/msg113575.html
> >>>>>>>> >>> >>
> >>>>>>>> >>> >>
> >>>>>>>> >>> >> Tha
, Dec 8, 2020 at 4:01 PM Omnia Ibrahim
>>> wrote:
>>>
>>>> Hi everyone,
>>>> I’m proposing a new KIP for MirrorMaker 2 to add the ability to control
>>>> internal topics naming convention. The proposal details are here
>>>> https:
find it here
>> https://www.mail-archive.com/dev@kafka.apache.org/msg113373.html
>>
>> On Tue, Dec 8, 2020 at 4:01 PM Omnia Ibrahim
>> wrote:
>>
>>> Hi everyone,
>>> I’m proposing a new KIP for MirrorMaker 2 to add the ability to control
>>> int
12:58 PM Mickael Maison <
>>>>>>>> mickael.mai...@gmail.com> wrote:
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> Hi Omnia,
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> Thank you for the reply, it makes sense.
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> A couple more comments:
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> 1) I'm assuming the new interface and default
>>>>>>>> implementation will be
>>>>>>>> >>> >>> in the mirror-client project? as the names of some of these
>>>>>>>> topics are
>>>>>>>> >>> >>> needed by RemoteClusterUtils on the client-side.
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> 2) I'm about to open a KIP to specify where the
>>>>>>>> offset-syncs topic is
>>>>>>>> >>> >>> created by MM2. In restricted environments, we'd prefer MM2
>>>>>>>> to only
>>>>>>>> >>> >>> have read access to the source cluster and have the
>>>>>>>> offset-syncs on
>>>>>>>> >>> >>> the target cluster. I think allowing to specify the cluster
>>>>>>>> where to
>>>>>>>> >>> >>> create that topic would be a natural extension of the
>>>>>>>> interface you
>>>>>>>> >>> >>> propose here.
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> So I wonder if your interface could be named
>>>>>>>> InternalTopicsPolicy?
>>>>>>>> >>> >>> That's a bit more generic than InternalTopicNamingPolicy.
>>>>>>>> That would
>>>>>>>> >>> >>> also match the configuration setting,
>>>>>>>> internal.topics.policy.class,
>>>>>>>> >>> >>> you're proposing.
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> Thanks
>>>>>>>> >>> >>>
>>>>>>>> >>> >>> On Thu, Dec 3, 2020 at 10:15 PM Omnia Ibrahim <
>>>>>>>> o.g.h.ibra...@gmail.com> wrote:
>>>>>>>> >>> >>> >
>>>>>>>> >>> >>> > Hi Mickael,
>>>>>>>> >>> >>> > Thanks for your feedback!
>>>>>>>> >>> >>> > Regards your question about having more configurations, I
>>>>>>>> considered adding
>>>>>>>> >>> >>> > configuration per each topic however this meant adding
>>>>>>>> more configurations
>>>>>>>> >>> >>> > for MM2 which already have so many, also the more
>>>>>>>> complicated and advanced
>>>>>>>> >>> >>> > replication pattern you have between clusters the more
>>>>>>>> configuration lines
>>>>>>>> >>> >>> > will be added to your MM2 config which isn't going to be
>>>>>>>> pretty if you
>>>>>>>> >>> >>> > don't have the same topics names across your clusters.
>>>>>>>> >>> >>> >
>>>>>>>> >>> >>> > Also, it added more complexity to the implementation as
>>>>>>>> MM2 need to
>>>>>>>> >>> >>> > 1- identify if a topic is checkpoints so we could list
>>>>>>>> the checkpoints
>>>>>>>> >>> >>> > topics in MirrorMaker 2 utils as one cluster could have X
>>>>>>>> numbers
>>>>>>>> >>> >>> > checkpoints topics if it's connected to X clusters, this
>>>>>>>> is done right now
>>>>>>>> >>> >>> > by listing any topic with suffix `.checkpoints.internal`.
>>>>>>>> This could be
>>>>>>>> >>> >>> > done by add `checkpoints.topic.suffix` config but this
>>>>>>>> would make an
>>>>>>>> >>> >>> > assumption that checkpoints will always have a suffix
>>>>>>>> also having a suffix
>>>>>>>> >>> >>> > means that we may need a separator as well to concatenate
>>>>>>>> this suffix with
>>>>>>>> >>> >>> > a prefix to identify source cluster name.
>>>>>>>> >>> >>> > 2- identify if a topic is internal, so it shouldn't be
>>>>>>>> replicated or track
>>>>>>>> >>> >>> > checkpoints for it, right now this is relaying on
>>>>>>>> disallow topics with
>>>>>>>> >>> >>> > `.internal` suffix to be not replicated and not tracked
>>>>>>>> in checkpoints but
>>>>>>>> >>> >>> > with making topics configurable we need a way to define
>>>>>>>> what is an internal
>>>>>>>> >>> >>> > topic. This could be done by making using a list of all
>>>>>>>> internal topics
>>>>>>>> >>> >>> > have been entered to the configuration.
>>>>>>>> >>> >>> >
>>>>>>>> >>> >>> > So having an interface seemed easier and also give more
>>>>>>>> flexibility for
>>>>>>>> >>> >>> > users to define their own topics name, define what is
>>>>>>>> internal topic means,
>>>>>>>> >>> >>> > how to find checkpoints topics and it will be one line
>>>>>>>> config for each
>>>>>>>> >>> >>> > herder, also it more consistence with MM2 code as MM2
>>>>>>>> config have
>>>>>>>> >>> >>> > TopicFilter, ReplicationPolicy, GroupFilter, etc as
>>>>>>>> interface and they can
>>>>>>>> >>> >>> > be overridden by providing a custom implementation for
>>>>>>>> them or have some
>>>>>>>> >>> >>> > config that change their default implementations.
>>>>>>>> >>> >>> >
>>>>>>>> >>> >>> > Hope this answer your question. I also updated the KIP to
>>>>>>>> add this to the
>>>>>>>> >>> >>> > rejected solutions.
>>>>>>>> >>> >>> >
>>>>>>>> >>> >>> >
>>>>>>>> >>> >>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison <
>>>>>>>> mickael.mai...@gmail.com>
>>>>>>>> >>> >>> > wrote:
>>>>>>>> >>> >>> >
>>>>>>>> >>> >>> > > Hi Omnia,
>>>>>>>> >>> >>> > >
>>>>>>>> >>> >>> > > Thanks for the KIP. Indeed being able to configure
>>>>>>>> MM2's internal
>>>>>>>> >>> >>> > > topic names would be a nice improvement.
>>>>>>>> >>> >>> > >
>>>>>>>> >>> >>> > > Looking at the KIP, I was surprised you propose an
>>>>>>>> interface to allow
>>>>>>>> >>> >>> > > users to specify names. Have you considered making
>>>>>>>> names changeable
>>>>>>>> >>> >>> > > via configurations? If so, we should definitely mention
>>>>>>>> it in the
>>>>>>>> >>> >>> > > rejected alternatives as it's the first method that
>>>>>>>> comes to mind.
>>>>>>>> >>> >>> > >
>>>>>>>> >>> >>> > > I understand an interface gives a lot of flexibility
>>>>>>>> but I'd expect
>>>>>>>> >>> >>> > > topic names to be relatively simple and known in
>>>>>>>> advance in most
>>>>>>>> >>> >>> > > cases.
>>>>>>>> >>> >>> > >
>>>>>>>> >>> >>> > > I've not checked all use cases but something like below
>>>>>>>> felt appropriate:
>>>>>>>> >>> >>> > > clusters = primary,backup
>>>>>>>> >>> >>> > >
>>>>>>>> primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
>>>>>>>> >>> >>> > >
>>>>>>>> >>> >>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim <
>>>>>>>> o.g.h.ibra...@gmail.com>
>>>>>>>> >>> >>> > > wrote:
>>>>>>>> >>> >>> > > >
>>>>>>>> >>> >>> > > > Hey everyone,
>>>>>>>> >>> >>> > > > Please take a look at KIP-690:
>>>>>>>> >>> >>> > > >
>>>>>>>> >>> >>> > > >
>>>>>>>> >>> >>> > >
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>>>>>>>> >>> >>> > > >
>>>>>>>> >>> >>> > > > Thanks for your feedback and support.
>>>>>>>> >>> >>> > > >
>>>>>>>> >>> >>> > > > Omnia
>>>>>>>> >>> >>> > > >
>>>>>>>> >>> >>> > >
>>>>>>>>
>>>>>>>
gt;>
>>> Thanks Omnia, makes sense to me.
>>>
>>> > Customers who have their customised ReplicationPolicy will need to add
>>> the definition of their internal topics naming convention
>>>
>>> I wonder should we include default impls in the int
t; customisation using `replication.policy.separator` and use the separator in
> the DefaultReplicationPolicy
>
> On Wed, Apr 28, 2021 at 1:31 AM Ryanne Dolan
> wrote:
>
>> Thanks Omnia, makes sense to me.
>>
>> > Customers who have their customised ReplicationPolicy will need to add
&g
8, 2021 at 1:31 AM Ryanne Dolan wrote:
> Thanks Omnia, makes sense to me.
>
> > Customers who have their customised ReplicationPolicy will need to add
> the definition of their internal topics naming convention
>
> I wonder should we include default impls in the interf
Thanks Omnia, makes sense to me.
> Customers who have their customised ReplicationPolicy will need to add
the definition of their internal topics naming convention
I wonder should we include default impls in the interface to avoid that
requirement?
Ryanne
On Sun, Apr 25, 2021, 2:20 PM Om
;>> >>> >>> > Also, it added more complexity to the implementation as MM2
>>> need to
>>> >>> >>> > 1- identify if a topic is checkpoints so we could list the
>>> checkpoints
>>> >>> >>> > topics in MirrorMaker 2 utils as one cluster could have X
>>> numbers
>>> >>> >>> > checkpoints topics if it's connected to X clusters, this is
>>> done right now
>>> >>> >>> > by listing any topic with suffix `.checkpoints.internal`. This
>>> could be
>>> >>> >>> > done by add `checkpoints.topic.suffix` config but this would
>>> make an
>>> >>> >>> > assumption that checkpoints will always have a suffix also
>>> having a suffix
>>> >>> >>> > means that we may need a separator as well to concatenate this
>>> suffix with
>>> >>> >>> > a prefix to identify source cluster name.
>>> >>> >>> > 2- identify if a topic is internal, so it shouldn't be
>>> replicated or track
>>> >>> >>> > checkpoints for it, right now this is relaying on disallow
>>> topics with
>>> >>> >>> > `.internal` suffix to be not replicated and not tracked in
>>> checkpoints but
>>> >>> >>> > with making topics configurable we need a way to define what
>>> is an internal
>>> >>> >>> > topic. This could be done by making using a list of all
>>> internal topics
>>> >>> >>> > have been entered to the configuration.
>>> >>> >>> >
>>> >>> >>> > So having an interface seemed easier and also give more
>>> flexibility for
>>> >>> >>> > users to define their own topics name, define what is internal
>>> topic means,
>>> >>> >>> > how to find checkpoints topics and it will be one line config
>>> for each
>>> >>> >>> > herder, also it more consistence with MM2 code as MM2 config
>>> have
>>> >>> >>> > TopicFilter, ReplicationPolicy, GroupFilter, etc as interface
>>> and they can
>>> >>> >>> > be overridden by providing a custom implementation for them or
>>> have some
>>> >>> >>> > config that change their default implementations.
>>> >>> >>> >
>>> >>> >>> > Hope this answer your question. I also updated the KIP to add
>>> this to the
>>> >>> >>> > rejected solutions.
>>> >>> >>> >
>>> >>> >>> >
>>> >>> >>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison <
>>> mickael.mai...@gmail.com>
>>> >>> >>> > wrote:
>>> >>> >>> >
>>> >>> >>> > > Hi Omnia,
>>> >>> >>> > >
>>> >>> >>> > > Thanks for the KIP. Indeed being able to configure MM2's
>>> internal
>>> >>> >>> > > topic names would be a nice improvement.
>>> >>> >>> > >
>>> >>> >>> > > Looking at the KIP, I was surprised you propose an interface
>>> to allow
>>> >>> >>> > > users to specify names. Have you considered making names
>>> changeable
>>> >>> >>> > > via configurations? If so, we should definitely mention it
>>> in the
>>> >>> >>> > > rejected alternatives as it's the first method that comes to
>>> mind.
>>> >>> >>> > >
>>> >>> >>> > > I understand an interface gives a lot of flexibility but I'd
>>> expect
>>> >>> >>> > > topic names to be relatively simple and known in advance in
>>> most
>>> >>> >>> > > cases.
>>> >>> >>> > >
>>> >>> >>> > > I've not checked all use cases but something like below felt
>>> appropriate:
>>> >>> >>> > > clusters = primary,backup
>>> >>> >>> > >
>>> primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
>>> >>> >>> > >
>>> >>> >>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim <
>>> o.g.h.ibra...@gmail.com>
>>> >>> >>> > > wrote:
>>> >>> >>> > > >
>>> >>> >>> > > > Hey everyone,
>>> >>> >>> > > > Please take a look at KIP-690:
>>> >>> >>> > > >
>>> >>> >>> > > >
>>> >>> >>> > >
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>>> >>> >>> > > >
>>> >>> >>> > > > Thanks for your feedback and support.
>>> >>> >>> > > >
>>> >>> >>> > > > Omnia
>>> >>> >>> > > >
>>> >>> >>> > >
>>>
>>
s.topic.suffix` config but this would
>> make an
>> >>> >>> > assumption that checkpoints will always have a suffix also
>> having a suffix
>> >>> >>> > means that we may need a separator as well to concatenate this
>> suffix with
>> >>> >>> > a prefix to identify source cluster name.
>> >>> >>> > 2- identify if a topic is internal, so it shouldn't be
>> replicated or track
>> >>> >>> > checkpoints for it, right now this is relaying on disallow
>> topics with
>> >>> >>> > `.internal` suffix to be not replicated and not tracked in
>> checkpoints but
>> >>> >>> > with making topics configurable we need a way to define what is
>> an internal
>> >>> >>> > topic. This could be done by making using a list of all
>> internal topics
>> >>> >>> > have been entered to the configuration.
>> >>> >>> >
>> >>> >>> > So having an interface seemed easier and also give more
>> flexibility for
>> >>> >>> > users to define their own topics name, define what is internal
>> topic means,
>> >>> >>> > how to find checkpoints topics and it will be one line config
>> for each
>> >>> >>> > herder, also it more consistence with MM2 code as MM2 config
>> have
>> >>> >>> > TopicFilter, ReplicationPolicy, GroupFilter, etc as interface
>> and they can
>> >>> >>> > be overridden by providing a custom implementation for them or
>> have some
>> >>> >>> > config that change their default implementations.
>> >>> >>> >
>> >>> >>> > Hope this answer your question. I also updated the KIP to add
>> this to the
>> >>> >>> > rejected solutions.
>> >>> >>> >
>> >>> >>> >
>> >>> >>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison <
>> mickael.mai...@gmail.com>
>> >>> >>> > wrote:
>> >>> >>> >
>> >>> >>> > > Hi Omnia,
>> >>> >>> > >
>> >>> >>> > > Thanks for the KIP. Indeed being able to configure MM2's
>> internal
>> >>> >>> > > topic names would be a nice improvement.
>> >>> >>> > >
>> >>> >>> > > Looking at the KIP, I was surprised you propose an interface
>> to allow
>> >>> >>> > > users to specify names. Have you considered making names
>> changeable
>> >>> >>> > > via configurations? If so, we should definitely mention it in
>> the
>> >>> >>> > > rejected alternatives as it's the first method that comes to
>> mind.
>> >>> >>> > >
>> >>> >>> > > I understand an interface gives a lot of flexibility but I'd
>> expect
>> >>> >>> > > topic names to be relatively simple and known in advance in
>> most
>> >>> >>> > > cases.
>> >>> >>> > >
>> >>> >>> > > I've not checked all use cases but something like below felt
>> appropriate:
>> >>> >>> > > clusters = primary,backup
>> >>> >>> > > primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
>> >>> >>> > >
>> >>> >>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim <
>> o.g.h.ibra...@gmail.com>
>> >>> >>> > > wrote:
>> >>> >>> > > >
>> >>> >>> > > > Hey everyone,
>> >>> >>> > > > Please take a look at KIP-690:
>> >>> >>> > > >
>> >>> >>> > > >
>> >>> >>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>> >>> >>> > > >
>> >>> >>> > > > Thanks for your feedback and support.
>> >>> >>> > > >
>> >>> >>> > > > Omnia
>> >>> >>> > > >
>> >>> >>> > >
>>
>
pics with
> >>> >>> > `.internal` suffix to be not replicated and not tracked in
> checkpoints but
> >>> >>> > with making topics configurable we need a way to define what is
> an internal
> >>> >>> > topic. This could be done by making using a list of all internal
> topics
> >>> >>> > have been entered to the configuration.
> >>> >>> >
> >>> >>> > So having an interface seemed easier and also give more
> flexibility for
> >>> >>> > users to define their own topics name, define what is internal
> topic means,
> >>> >>> > how to find checkpoints topics and it will be one line config
> for each
> >>> >>> > herder, also it more consistence with MM2 code as MM2 config have
> >>> >>> > TopicFilter, ReplicationPolicy, GroupFilter, etc as interface
> and they can
> >>> >>> > be overridden by providing a custom implementation for them or
> have some
> >>> >>> > config that change their default implementations.
> >>> >>> >
> >>> >>> > Hope this answer your question. I also updated the KIP to add
> this to the
> >>> >>> > rejected solutions.
> >>> >>> >
> >>> >>> >
> >>> >>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison <
> mickael.mai...@gmail.com>
> >>> >>> > wrote:
> >>> >>> >
> >>> >>> > > Hi Omnia,
> >>> >>> > >
> >>> >>> > > Thanks for the KIP. Indeed being able to configure MM2's
> internal
> >>> >>> > > topic names would be a nice improvement.
> >>> >>> > >
> >>> >>> > > Looking at the KIP, I was surprised you propose an interface
> to allow
> >>> >>> > > users to specify names. Have you considered making names
> changeable
> >>> >>> > > via configurations? If so, we should definitely mention it in
> the
> >>> >>> > > rejected alternatives as it's the first method that comes to
> mind.
> >>> >>> > >
> >>> >>> > > I understand an interface gives a lot of flexibility but I'd
> expect
> >>> >>> > > topic names to be relatively simple and known in advance in
> most
> >>> >>> > > cases.
> >>> >>> > >
> >>> >>> > > I've not checked all use cases but something like below felt
> appropriate:
> >>> >>> > > clusters = primary,backup
> >>> >>> > > primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
> >>> >>> > >
> >>> >>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com>
> >>> >>> > > wrote:
> >>> >>> > > >
> >>> >>> > > > Hey everyone,
> >>> >>> > > > Please take a look at KIP-690:
> >>> >>> > > >
> >>> >>> > > >
> >>> >>> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> >>> >>> > > >
> >>> >>> > > > Thanks for your feedback and support.
> >>> >>> > > >
> >>> >>> > > > Omnia
> >>> >>> > > >
> >>> >>> > >
>
nal, so it shouldn't be
> replicated or track
> >>> >>> > checkpoints for it, right now this is relaying on disallow
> topics with
> >>> >>> > `.internal` suffix to be not replicated and not tracked in
> checkpoints but
> >>> &g
>>> >>> > with making topics configurable we need a way to define what is an
>>> >>> > internal
>>> >>> > topic. This could be done by making using a list of all internal
>>> >>> > topics
>>> >>> > have been entered to the configuration.
>>> >>> >
>>> >>> > So having an interface seemed easier and also give more flexibility
>>> >>> > for
>>> >>> > users to define their own topics name, define what is internal topic
>>> >>> > means,
>>> >>> > how to find checkpoints topics and it will be one line config for each
>>> >>> > herder, also it more consistence with MM2 code as MM2 config have
>>> >>> > TopicFilter, ReplicationPolicy, GroupFilter, etc as interface and
>>> >>> > they can
>>> >>> > be overridden by providing a custom implementation for them or have
>>> >>> > some
>>> >>> > config that change their default implementations.
>>> >>> >
>>> >>> > Hope this answer your question. I also updated the KIP to add this to
>>> >>> > the
>>> >>> > rejected solutions.
>>> >>> >
>>> >>> >
>>> >>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison
>>> >>> >
>>> >>> > wrote:
>>> >>> >
>>> >>> > > Hi Omnia,
>>> >>> > >
>>> >>> > > Thanks for the KIP. Indeed being able to configure MM2's internal
>>> >>> > > topic names would be a nice improvement.
>>> >>> > >
>>> >>> > > Looking at the KIP, I was surprised you propose an interface to
>>> >>> > > allow
>>> >>> > > users to specify names. Have you considered making names changeable
>>> >>> > > via configurations? If so, we should definitely mention it in the
>>> >>> > > rejected alternatives as it's the first method that comes to mind.
>>> >>> > >
>>> >>> > > I understand an interface gives a lot of flexibility but I'd expect
>>> >>> > > topic names to be relatively simple and known in advance in most
>>> >>> > > cases.
>>> >>> > >
>>> >>> > > I've not checked all use cases but something like below felt
>>> >>> > > appropriate:
>>> >>> > > clusters = primary,backup
>>> >>> > > primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
>>> >>> > >
>>> >>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim
>>> >>> > >
>>> >>> > > wrote:
>>> >>> > > >
>>> >>> > > > Hey everyone,
>>> >>> > > > Please take a look at KIP-690:
>>> >>> > > >
>>> >>> > > >
>>> >>> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>>> >>> > > >
>>> >>> > > > Thanks for your feedback and support.
>>> >>> > > >
>>> >>> > > > Omnia
>>> >>> > > >
>>> >>> > >
seemed easier and also give more flexibility
>> for
>> >>> > users to define their own topics name, define what is internal
>> topic means,
>> >>> > how to find checkpoints topics and it will be one line config for
>> each
>> >>>
custom implementation for them or have
> some
> >>> > config that change their default implementations.
> >>> >
> >>> > Hope this answer your question. I also updated the KIP to add this
> to the
> >>> > rejected solutions.
> >>> >
> >>> >
> >>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison <
> mickael.mai...@gmail.com>
> >>> > wrote:
> >>> >
> >>> > > Hi Omnia,
> >>> > >
> >>> > > Thanks for the KIP. Indeed being able to configure MM2's internal
> >>> > > topic names would be a nice improvement.
> >>> > >
> >>> > > Looking at the KIP, I was surprised you propose an interface to
> allow
> >>> > > users to specify names. Have you considered making names changeable
> >>> > > via configurations? If so, we should definitely mention it in the
> >>> > > rejected alternatives as it's the first method that comes to mind.
> >>> > >
> >>> > > I understand an interface gives a lot of flexibility but I'd expect
> >>> > > topic names to be relatively simple and known in advance in most
> >>> > > cases.
> >>> > >
> >>> > > I've not checked all use cases but something like below felt
> appropriate:
> >>> > > clusters = primary,backup
> >>> > > primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
> >>> > >
> >>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim <
> o.g.h.ibra...@gmail.com>
> >>> > > wrote:
> >>> > > >
> >>> > > > Hey everyone,
> >>> > > > Please take a look at KIP-690:
> >>> > > >
> >>> > > >
> >>> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> >>> > > >
> >>> > > > Thanks for your feedback and support.
> >>> > > >
> >>> > > > Omnia
> >>> > > >
> >>> > >
>
;>> > rejected solutions.
>>> >
>>> >
>>> > On Thu, Dec 3, 2020 at 3:19 PM Mickael Maison
>>> > wrote:
>>> >
>>> > > Hi Omnia,
>>> > >
>>> > > Thanks for the KIP. Indeed being ab
mnia Ibrahim
> wrote:
>
>> Hi everyone,
>> I’m proposing a new KIP for MirrorMaker 2 to add the ability to control
>> internal topics naming convention. The proposal details are here
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuratio
.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>> >
>> > Thanks for your time and feedback.
>> >
>> > Omnia
>> >
>>
>
KIP, I was surprised you propose an interface to allow
>> > > users to specify names. Have you considered making names changeable
>> > > via configurations? If so, we should definitely mention it in the
>> > > rejected alternatives as it's the first method that comes to mind.
>> > >
>> > > I understand an interface gives a lot of flexibility but I'd expect
>> > > topic names to be relatively simple and known in advance in most
>> > > cases.
>> > >
>> > > I've not checked all use cases but something like below felt
>> appropriate:
>> > > clusters = primary,backup
>> > > primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
>> > >
>> > > On Tue, Dec 1, 2020 at 3:36 PM Omnia Ibrahim > >
>> > > wrote:
>> > > >
>> > > > Hey everyone,
>> > > > Please take a look at KIP-690:
>> > > >
>> > > >
>> > >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>> > > >
>> > > > Thanks for your feedback and support.
>> > > >
>> > > > Omnia
>> > > >
>> > >
>>
>
al topics naming convention. The proposal details are here
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>
> Please vote in this thread.
> Thanks
> Omnia
>
configurations? If so, we should definitely mention it in the
> > > rejected alternatives as it's the first method that comes to mind.
> > >
> > > I understand an interface gives a lot of flexibility but I'd expect
> > > topic names to be relatively simple
y simple and known in advance in most
> > cases.
> >
> > I've not checked all use cases but something like below felt appropriate:
> > clusters = primary,backup
> > primary->backup.offsets-sync.topic=backup.mytopic-offsets-sync
> >
> > On Tue,
figuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> >
> > Thanks for your time and feedback.
> >
> > Omnia
> >
>
Hi everyone,
I’m proposing a new KIP for MirrorMaker 2 to add the ability to control
internal topics naming convention. The proposal details are here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
P-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>
> Thanks for your time and feedback.
>
> Omnia
>
PM Omnia Ibrahim
> wrote:
> >
> > Hey everyone,
> > Please take a look at KIP-690:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
> >
> > Thanks for your feedback and support.
> >
> > Omnia
> >
>
690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
>
> Thanks for your feedback and support.
>
> Omnia
>
Hey everyone,
Please take a look at KIP-690:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
Thanks for your feedback and support.
Omnia
Hey everyone,
Please take a look at KIP-690
https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
Thanks for the feedback and support
Omnia
Hi everyone
I want to start discussion of the KIP 690, the proposal is here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
Thanks for your time and feedback.
Omnia
Hi everyone
I want to start discussion of the KIP 690, the proposal is here
https://cwiki.apache.org/confluence/display/KAFKA/KIP-690%3A+Add+additional+configuration+to+control+MirrorMaker+2+internal+topics+naming+convention
Thanks for your time and feedback.
Omnia
Omnia Ibrahim created KAFKA-10777:
-
Summary: Add additional configuration to control MM2 internal
topics naming convention
Key: KAFKA-10777
URL: https://issues.apache.org/jira/browse/KAFKA-10777
If the schema is no longer compatible I think it makes sense to create a
new topic.
Offsets wise you can just have the consumer finish consuming from the old
topic/schema (i.e. migrate all producers to use the new schema format
first) and then migrate consumers to use the new schema format (and th
Hello,
I'm here to get some advice regarding Kafka topics naming.
If I name my topic "users" and I publish Avro events with a schema registry
using a given compatibility mode.
One day, due to business reasons, I have to break my schema, what would you
do?
add a new topic (let's call it "users-v1
54 matches
Mail list logo
