Thanks again Ryanne, I didn't realize that MM2 would handle that. However, I'm unable to mirror the remote topic back to the source cluster by adding it to the topic whitelist. I've also tried to update the topic blacklist and remove ".*\.replica" (since the blacklists take precedence over the whitelists), but that doesn't seem to be doing much either? Is there something else I should be aware of in the mm2.properties file?
Appreciate all your help! Josh On Wed, Aug 19, 2020 at 12:55 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Josh, if you have two clusters with bidirectional replication, you only get > two copies of each record. MM2 won't replicate the data "upstream", cuz it > knows it's already there. In particular, MM2 knows not to create topics > like B.A.topic1 on cluster A, as this would be an unnecessary cycle. > > > is there a reason for MM2 not emitting checkpoint data for the source > topic AND the remote topic > > No, not really! I think it would be surprising if one-directional flows > insisted on writing checkpoints both ways -- but it's also surprising that > you need to explicitly allow a remote topic to be checkpointed. I'd support > changing this, fwiw. > > Ryanne > > On Wed, Aug 19, 2020 at 2:30 PM Josh C <josh.fl.c...@gmail.com> wrote: > > > Sorry, correction -- I am realizing now it would be 3 copies of the same > > topic data as A.topic1 has different data than B.topic1. However, that > > would still be 3 copies as opposed to just 2 with something like topic1 > and > > A.topic1. > > > > As well, if I were to explicitly replicate the remote topic back to the > > source cluster by adding it to the topic whitelist, would I also need to > > update the topic blacklist and remove ".*\.replica" (since the blacklists > > take precedence over the whitelists)? > > > > Josh > > > > On Wed, Aug 19, 2020 at 11:46 AM Josh C <josh.fl.c...@gmail.com> wrote: > > > > > Thanks for the clarification Ryanne. In the context of active/active > > > clusters, does this mean there would be 6 copies of the same topic > data? > > > > > > A topics: > > > - topic1 > > > - B.topic1 > > > - B.A.topic1 > > > > > > B topics: > > > - topic1 > > > - A.topic1 > > > - A.B.topic1 > > > > > > Out of curiosity, is there a reason for MM2 not emitting checkpoint > data > > > for the source topic AND the remote topic as a pair as opposed to > having > > to > > > explicitly replicate the remote topic back to the source cluster just > to > > > have the checkpoints emitted upstream? > > > > > > Josh > > > > > > On Wed, Aug 19, 2020 at 6:16 AM Ryanne Dolan <ryannedo...@gmail.com> > > > wrote: > > > > > >> Josh, yes it's possible to migrate the consumer group back to the > source > > >> topic, but you need to explicitly replicate the remote topic back to > the > > >> source cluster -- otherwise no checkpoints will flow "upstream": > > >> > > >> A->B.topics=test1 > > >> B->A.topics=A.test1 > > >> > > >> After the first checkpoint is emitted upstream, > > >> RemoteClusterUtils.translateOffsets() will translate B's A.test1 > offsets > > >> into A's test1 offsets for you. > > >> > > >> Ryanne > > >> > > >> On Tue, Aug 18, 2020 at 5:56 PM Josh C <josh.fl.c...@gmail.com> > wrote: > > >> > > >> > Hi there, > > >> > > > >> > I'm currently exploring MM2 and having some trouble with the > > >> > RemoteClusterUtils.translateOffsets() method. I have been successful > > in > > >> > migrating a consumer group from the source cluster to the target > > >> cluster, > > >> > but was wondering how I could migrate this consumer group back to > the > > >> > original source topic? > > >> > > > >> > It is my understanding that there isn't any checkpoint data being > > >> > emitted for this consumer group since it is consuming from a > mirrored > > >> topic > > >> > in the target cluster. I'm currently getting an empty map since > there > > >> isn't > > >> > any checkpoint data for 'target.checkpoints.internal' in the source > > >> > cluster. So, I was wondering how would I get these new translated > > >> offsets > > >> > to migrate the consumer group back to the source cluster? > > >> > > > >> > Please let me know if my question was unclear or if you require > > further > > >> > clarification! Appreciate the help. > > >> > > > >> > Thanks, > > >> > Josh > > >> > > > >> > > > > > >
