Thanks, Jamie:
I tried, but still not working☹, here is the updated properties file:
clusters=CDL,PC
CDL.bootstrap.servers=xxx
PC.bootstrap.servers=xx
PC.security.protocol=SSL
consumer.auto.offset.reset=earliest
group.id=test-mm2-2
CDL->PC.enabled=true
topics= .*
groups= .*
topic.blacklist= .*[\~\.]internal, .*\.replica, __.*,
heartbeats,nas\.vantage.*,INTEGRATION.*, ADPRM\..*, ADP_LIFION_INTEGRATION\..*,
AES\..*, BENEFITS\..*, DMX\..*, NASBILLING\..*, TAAS\..*
groups.blacklist= console-consumer-.*, connect-.*, __.*
tasks.max= 10
#emit.checkpoints.enabled= true
#sync.group.offsets.enabled= true
#offset.lag.max= 9223372036854775807
replication.factor= 3
checkpoints.topic.replication.factor= 3
heartbeats.topic.replication.factor= 3
offset-syncs.topic.replication.factor= 3
offset.storage.replication.factor= 3
status.storage.replication.factor= 3
config.storage.replication.factor= 3
transaction.state.log.replication.factor= 3
On 6/13/22, 4:25 PM, "Jamie" <jamied...@aol.co.uk.INVALID> wrote:
WARNING: Do not click links or open attachments unless you recognize the
source of the email and know the contents are safe.
**********************************************************************
Hi An,
Try setting consumer.auto.offset.reset=earliest
This will mean if the consumers in the MM2 tasks don’t have an offset for a
topic partition they will start at the beginning.
If you’ve already enabled the below you might have to use a new group.id so
that the consumers no longer have an offset that is valid. Since when you
stated MM2 with below config it will now have an offset stored in the offset
storage topic for each topic partition being consumed and therefore the
auto.offset.reset config won’t be applied.
Thanks,
Jamie
Sent from the all-new AOL app for iOS
On Tuesday, June 14, 2022, 12:19 am, An, Hongguo (CORP)
<hongguo...@adp.com.INVALID> wrote:
Hi:
I am trying to use MM2 ( 3.3.0) to migration Kafka to a new server, but
looks like it only copys new data, any existing data is not copied.
Here is the connection properties:
clusters=CDL,PC
CDL.bootstrap.servers=xxx
PC.bootstrap.servers=xxx
PC.security.protocol=SSL
CDL->PC.enabled=true
topics= .*
groups= .*
topic.blacklist= .*[\~\.]internal, .*\.replica, __.*,
heartbeats,nas\.vantage.*,INTEGRATION.*, ADPRM\..*, ADP_LIFION_INTEGRATION\..*,
AES\..*, BENEFITS\..*, DMX\..*, NASBILLING\..*, TAAS\..*
groups.blacklist= console-consumer-.*, connect-.*, __.*
tasks.max= 10
#emit.checkpoints.enabled= true
#sync.group.offsets.enabled= true
#offset.lag.max= 9223372036854775807
replication.factor= 3
checkpoints.topic.replication.factor= 3
heartbeats.topic.replication.factor= 3
offset-syncs.topic.replication.factor= 3
offset.storage.replication.factor= 3
status.storage.replication.factor= 3
config.storage.replication.factor= 3
transaction.state.log.replication.factor= 3
Thanks in advance for any help
This message and any attachments are intended only for the use of the
addressee and may contain information that is privileged and confidential. If
the reader of the message is not the intended recipient or an authorized
representative of the intended recipient, you are hereby notified that any
dissemination of this communication is strictly prohibited. If you have
received this communication in error, notify the sender immediately by return
email and delete the message and any attachments from your system.
