Hi Ryanne, Thanks for the information regarding the manual translation approach. But for 2.7.0, I have a question. I thought this translation would happen implicitly. I saw test testOneWayReplicationWithAutoOffsetSync of MirrorConnectorsIntegrationTest that is supposed to test this behavior. But I have an impression that the consumers can't be migrated when there are some messages left in the original cluster. I posted this question to users@kafka a while ago but I didn't receive any responses. https://lists.apache.org/thread.html/r0728e9ea89b2713865a33396f108027ac3f0949c2496583532ee963c%40%3Cusers.kafka.apache.org%3E Could you or someone comment on this? Thanks. Regards, aki
El sáb, 2 ene 2021 a las 20:31, Ryanne Dolan (<ryannedo...@gmail.com>) escribió: > > Aki, that's right. Prior to 2.7, you can use the translateOffsets method > from within your client code, or you can write a little command-line tool > to do it. I've done the latter for Cloudera's srm-control tool, as > documented here: > https://docs.cloudera.com/csp/2.0.1/srm-using/topics/srm-migrating-consumer-groups.html > > Ryanne > > On Thu, Dec 17, 2020, 5:23 PM Aki Yoshida <elak...@gmail.com> wrote: > > > I'm answering to my own mail and would like to hear if this assumption > > is correct. > > it looks like I need 2.7.0 to have the automatic translation. > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0 > > for earlier versions, there is no way to use the console client but > > use java API RemoteClusterUtils.translateOffsets() to get the offset > > and re-consuming from the new topic. > > Is this correct? > > > > El jue, 17 dic 2020 a las 21:18, Aki Yoshida (<elak...@gmail.com>) > > escribió: > > > > > > Hi, > > > I have a question regarding how to migrate a consumer when the > > > subscribing topic has been migrated to the target Kafka broker. > > > Suppose a consumer is consuming from topic1 at the source Kafka > > > broker. When MM2 mirrors this topic using options > > > source.cluster.alias="", replication.policy.separator="", topic named > > > topic1 shows up at the target Kafka broker. > > > > > > When the consumer instance simply switches the bootstrap server to > > > start consuming from this topic1 at the target broker, the consumer > > > seems to start subscribing from the latest offset. I thought the > > > consumer offsets were also migrated to the target and the consumer > > > could simply start subscribing from the mirrored topic from the > > > continued offset. Did I miss some configuration parameters or does the > > > consumer need to perform some actions to be able to consume the > > > records seamlessly? > > > > > > I appreciate for your help. Thanks. > > > > > > regards, aki > >
