hello Aki, I guess overall you are right.
When the consumer have not finish consuming the latest message from primary and switch to the backup, the left messages will be still replicated from primary to backup by MM, so your consumer will still be able to consume the left messages at backup, this is what your commit illustrated. I think the original test was correct as well, since it assumes no left message in primary before switching consumer to backup. If you are interested in contributing your commit, it would be great to submit it on https://github.com/apache/kafka/pull/9224, as MM tests could be significantly refactored soon. Thanks On 2021/01/13 08:14:06, Aki Yoshida <elak...@gmail.com> wrote: > Hi Ning, > I mean by "when there are some messages left in the original cluster", > when the consumer switches from the original cluster to the new > cluster before entirely consuming the messages at the original > cluster. > This commit illustrates this situation. > https://github.com/elakito/kafka/commit/9063db0f8c4a53f5d9764612af898981d499a7b7 > > regards, aki > > El mié, 13 ene 2021 a las 3:17, Ning Zhang (<ning2008w...@gmail.com>) > escribió: > > > > Hello Aki, > > > > Can you elaborate on "when there are some messages left in the original > > cluster" ? > > > > On 2021/01/12 13:01:40, Aki Yoshida <elak...@gmail.com> wrote: > > > Hi Ryanne, > > > Thanks for the information regarding the manual translation approach. > > > But for 2.7.0, I have a question. I thought this translation would > > > happen implicitly. > > > I saw test testOneWayReplicationWithAutoOffsetSync of > > > MirrorConnectorsIntegrationTest that is supposed to test this > > > behavior. But I have an impression that the consumers can't be > > > migrated when there are some messages left in the original cluster. I > > > posted this question to users@kafka a while ago but I didn't receive > > > any responses. > > > https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E > > > Could you or someone comment on this? > > > Thanks. > > > Regards, aki > > > > > > El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ryannedo...@gmail.com>) > > > escribió: > > > > > > > > Aki, that's right. Prior to 2.7, you can use the translateOffsets method > > > > from within your client code, or you can write a little command-line > > > > tool > > > > to do it. I've done the latter for Cloudera's srm-control tool, as > > > > documented here: > > > > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html > > > > > > > > Ryanne > > > > > > > > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <elak...@gmail.com> wrote: > > > > > > > > > I'm answering to my own mail and would like to hear if this assumption > > > > > is correct. > > > > > it looks like I need 2.7.0 to have the automatic translation. > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0 > > > > > for earlier versions, there is no way to use the console client but > > > > > use java API RemoteClusterUtils.translateOffsets() to get the offset > > > > > and re-consuming from the new topic. > > > > > Is this correct? > > > > > > > > > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<elak...@gmail.com>) > > > > > escribió: > > > > > > > > > > > > Hi, > > > > > > I have a question regarding how to migrate a consumer when the > > > > > > subscribing topic has been migrated to the target Kafka broker. > > > > > > Suppose a consumer is consuming from topic1 at the source Kafka > > > > > > broker. When MM2 mirrors this topic using options > > > > > > source.cluster.alias="", replication.policy.separator="", topic > > > > > > named > > > > > > topic1 shows up at the target Kafka broker. > > > > > > > > > > > > When the consumer instance simply switches the bootstrap server to > > > > > > start consuming from this topic1 at the target broker, the consumer > > > > > > seems to start subscribing from the latest offset. I thought the > > > > > > consumer offsets were also migrated to the target and the consumer > > > > > > could simply start subscribing from the mirrored topic from the > > > > > > continued offset. Did I miss some configuration parameters or does > > > > > > the > > > > > > consumer need to perform some actions to be able to consume the > > > > > > records seamlessly? > > > > > > > > > > > > I appreciate for your help. Thanks. > > > > > > > > > > > > regards, aki > > > > > > > > >
