; > > > > Thanks,
> > > > > > Daniel
> > > > > >
> > > > > > Vidor Kanalas ezt írta (időpont: 2024.
> > > nov.
> > > > > > 15., P, 10:36):
> > > > > >
> > > > > >> Hi Daniel,
&g
bout. Suppose we have
> > a
> > > > >> topic on a prod cluster A that is replicated to a backup cluster B.
> > > > There
> > > > >> is a CG that is working through the messages on the backup cluster,
> > > > before
> > > > &
t is working through the messages on the backup cluster,
> > > before
> > > >> it’s promoted to the prod cluster. In this case that CG does not
> exist
> > > on
> > > >> cluster A, and it won’t be checkpointed (obviously), but it’s not
>
the above setup is an actual real-world use case,
> but
> > >> if it is, we need to make sure that CGs can get reverse checkpointed
> > even
> > >> if they initially don’t exist on the cluster. (it’s not a traditional
> > >> failover + failback scenario). Th
if it will be reverse checkpointed.
> >> I’m not certain if the above setup is an actual real-world use case, but
> >> if it is, we need to make sure that CGs can get reverse checkpointed
> even
> >> if they initially don’t exist on the cluster. (it’s not a traditional
&g
ing that a reverse
>> checkpointing group filter could be useful, but I agree that the same can
>> be achieved with the existing filter.
>>
>> Best,
>> Vidor
>>
>>
>> From: Dániel Urbán
>> Date: Friday, 15 November 2024 at 09:35
>> To: de
ree that the same can
> be achieved with the existing filter.
>
> Best,
> Vidor
>
>
> From: Dániel Urbán
> Date: Friday, 15 November 2024 at 09:35
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-1098: Reverse Checkpointing in MirrorMaker
> Hi Viktor,
>
>
agree that the same can be achieved with
the existing filter.
Best,
Vidor
From: Dániel Urbán
Date: Friday, 15 November 2024 at 09:35
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-1098: Reverse Checkpointing in MirrorMaker
Hi Viktor,
SVV3. In the current proposal, if the TopicFilter is
Hi Viktor,
SVV3. In the current proposal, if the TopicFilter is not provided, it
enables a different logic for reverse checkpointing - the task relies on
the ReplicationPolicy to detect if a topic in the source cluster is a
replica originating from the target cluster, e.g. in the A->B flow, the
to
Hi Daniel,
SVV3. Kind of an implementation detail. So I think using TopicFilter is
good, however I was wondering if we should provide a default implementation
instead of null? We have to implement the pass-through behavior anyways,
and it makes sense to me to do it in a filter.
SVV4. Also, an alte
Gentle bump - any comments are welcome.
This could fill an important gap in MM2, and would be nice to fix.
TIA
Daniel
Dániel Urbán ezt írta (időpont: 2024. nov. 4., H,
11:00):
> Hi Vidor,
>
> Thank you for your comments!
>
> 1. I think the optimization sounds nice, but would not work well with
>
Hi Vidor,
Thank you for your comments!
1. I think the optimization sounds nice, but would not work well with
TopicFilter implementations which can be dynamically updated in the
background (without touching the Connector configuration). If we actually
dropped the offset sync records for a topic wh
Hi Daniel,
This would indeed greatly reduce the duplicate processing on failbacks.
Few questions:
1. Since having a second offset-sync store can be memory intensive,
would it make sense to filter the topics in it based on the
reverseCheckpointingTopicFilter?
2. Would it make sens
Hi Daniel,
This would indeed greatly reduce the duplicate processing on failbacks.
Few questions:
1. Adding another offset sync store might be memory intensive. Would it make
sense (if possible) to filter the topics in it based on the
reverseCheckpointingTopicFilter?
2. Would it make sen
Hi Daniel,
This would indeed greatly reduce the duplicate processing on failbacks.
Few questions:
Adding another offset sync store might be memory intensive. Would it make sense
(if possible) to filter the topics in it based on the
reverseCheckpointingTopicFilter?
Would it make sense to add a
Hi Daniel,
This would indeed greatly reduce the duplicate processing on failbacks.
Few questions:
1. Adding another offset sync store might be memory intensive. Would it make
sense (if possible) to filter the topics in it based on the
reverseCheckpointingTopicFilter?
2. Would it make sen
Hi Viktor,
SVV1. Not easy to provide a number, but yes, it does scale with the number
of replicated topic partitions. Enabling this feature will add the overhead
of an extra consumer, and allocates memory for an offset-sync index for
each partition. The index is limited to 64 entries. I could give
Hi Daniel,
SVV1. Fair points about the performance impact. The next question is that
can we quantify it somehow, ie. does it scale with the number of topics to
reverse checkpoints, the offsets emitted, etc.?
I'll do one more pass on the KIP in the following days but I wanted to
reply to you with
Hi,
One more update. As I was working on the PR, I realized that the only way
to support IdentityReplicationPolicy is to add an extra topic filter to the
checkpointing. I updated the KIP accordingly.
I also opened a draft PR to demonstrate the proposed changes:
https://github.com/apache/kafka/pull
Hi all,
Just a bump/minor update here:
As I've started working on a POC of the proposed solution, I've realised
that the hard requirement related to the ReplicationPolicy implementation
can be eliminated, updated the KIP accordingly.
Daniel
Dániel Urbán ezt írta (időpont: 2024. okt. 21., H,
16:18
Hi Mickael,
Good point, I renamed the KIP and this thread:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1098%3A+Reverse+Checkpointing+in+MirrorMaker
Thank you,
Daniel
Mickael Maison ezt írta (időpont: 2024. okt.
21., H, 15:22):
> Hi Daniel,
>
> I've not had time to take a close look at
21 matches
Mail list logo
