hi,
I try to use mirror maker to replicate the data, and I have one trivial
question:
Do I need access to / add **all** Kafka Brokers ?
I have a 3.1 Cluster with 7 brokers and a new 3.5 with 5 brokers and I
want to migrate from the 3.1 (Zoo) to 3.5 (KRaft). Both clusters are in
different networks with a lot of firewalls and we allowed only access to
one of the old clusters:
My mirror-mager is on the end.
The problem is .. that if I start the sync .. it seems to be stuck
somewhere:
======================================
(org.apache.kafka.connect.mirror.MirrorHeartbeatConfig:369)
[2023-10-02 14:07:50,091] INFO [MirrorHeartbeatConnector|task-0]
WorkerSourceTask{id=MirrorHeartbeatConnector-0} Source task finished
initialization and start
(org.apache.kafka.connect.runtime.AbstractWorkerSource
Task:275)
[2023-10-02 14:07:50,113] INFO [MirrorHeartbeatConnector|task-0]
[Producer clientId=connector-producer-MirrorHeartbeatConnector-0]
Cluster ID: Rnpnd4EcRBeWo8vUrWlOIQ (org.apache.kafka.clients.Metadata:287)
[2023-10-02 14:07:50,238] INFO [MirrorSourceConnector|worker]
[AdminClient
clientId=FC-R02->FC-R01|MirrorSourceConnector|replication-source-admin]
Disconnecting from node 7 due to socket connection setup timeout.
The timeout value is 11336 ms. (org.apache.kafka.clients.NetworkClient:858)
[2023-10-02 14:07:50,828] INFO [MirrorCheckpointConnector|worker]
[AdminClient
clientId=FC-R02->FC-R01|MirrorCheckpointConnector|checkpoint-source-admin]
Disconnecting from node 7 due to socket connection setup ti
meout. The timeout value is 11943 ms.
(org.apache.kafka.clients.NetworkClient:858)
[2023-10-02 14:07:58,045] INFO [MirrorCheckpointConnector|worker]
[AdminClient
clientId=FC-R02->FC-R01|MirrorCheckpointConnector|checkpoint-source-admin]
Disconnecting from node 6 due to socket connection setup ti
meout. The timeout value is 8916 ms.
(org.apache.kafka.clients.NetworkClient:858)
[2023-10-02 14:08:00,271] INFO [MirrorSourceConnector|worker]
[AdminClient
clientId=FC-R02->FC-R01|MirrorSourceConnector|replication-source-admin]
Disconnecting from node 6 due to socket connection setup timeout.
The timeout value is 10018 ms. (org.apache.kafka.clients.NetworkClient:858)
[2023-10-02 14:08:00,359] INFO [MirrorSourceConnector|worker]
[AdminClient
clientId=FC-R02->FC-R01|MirrorSourceConnector|replication-source-admin]
Disconnecting from node 7 due to socket connection setup timeout.
The timeout value is 11266 ms. (org.apache.kafka.clients.NetworkClient:858)
[2023-10-02 14:08:00,440] INFO [MirrorSourceConnector|worker] loading
initial set of topic-partitions took 11359 ms
(org.apache.kafka.connect.mirror.Scheduler:95)
[2023-10-02 14:08:00,448] INFO [MirrorSourceConnector|worker] creating
downstream topic-partitions took 7 ms
(org.apache.kafka.connect.mirror.Scheduler:95)
[2023-10-02 14:08:00,454] INFO [MirrorSourceConnector|worker] refreshing
known target topics took 5 ms
(org.apache.kafka.connect.mirror.Scheduler:95)
[2023-10-02 14:08:00,457] INFO [MirrorSourceConnector|worker] Started
MirrorSourceConnector with 0 topic-partitions.
(org.apache.kafka.connect.mirror.MirrorSourceConnector:172)
[2023-10-02 14:08:00,458] INFO [MirrorSourceConnector|worker] Starting
MirrorSourceConnector took 11510 ms.
(org.apache.kafka.connect.mirror.MirrorSourceConnector:173)
[2023-10-02 14:08:00,461] INFO SourceConnectorConfig values:
config.action.reload = restart
connector.class =
org.apache.kafka.connect.mirror.MirrorSourceConnector
======================================
So, If I see it correct, it tries to connect to the other nodes .. which
is not allowed yet .. right ?
cu denny
#######################
clusters = FC-R02, FC-R01
FC-R02.bootstrap.servers = fc-r02-kafka-01.example.com:9093
FC-R01.bootstrap.servers = fc-r01-kafka-01.example.com:9092
FC-R02->FC-R01.enabled = true
FC-R01->FC-R02.enabled = false
FC-R02->FC-R01.topics =
"(blacklist_transaction.*|foo_transaction.*|chargebacks.*)"
replication.factor=4
sync.topic.acls.enabled=false
checkpoints.topic.replication.factor=3
heartbeats.topic.replication.factor=3
offset-syncs.topic.replication.factor=3
offset.storage.replication.factor=3
status.storage.replication.factor=3
config.storage.replication.factor=3
security.protocol: SASL_SSL
ssl.truststore.type = PEM
ssl.truststore.location=/etc/ssl/certs/ca-certificates.crt
ssl.enabled.protocols=TLSv1.2
security.protocol = SASL_SSL
FC-R01.sasl.mechanism=PLAIN
FC-R01.ssl.endpoint.identification.algorithm=https
FC-R01.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required \
serviceName='kafka' \
username='kafka' \
password='secret';
FC-R02.security.protocol: SSL
FC-R02.sasl.mechanism=PLAIN
FC-R02.ssl.endpoint.identification.algorithm=https
FC-R02.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
required \
serviceName='kafka' \
username='kafka' \
password='secret';
==========================================
