Dear all,
I’m stuck in getting a setup to work where MirrorMaker consumes from a
Confluent Kafka ensemble and produces to Azure Event Hub.
The MirrorMaker is started as a service like
/etc/systemd/system/kafka-azure-event-hub-mm.service
[Unit]
Description=Kafka Azure-Event-Hub MM Service
After=syslog.target network.target
[Service]
ExecStart=/opt/kafka_2.13-2.8.0/bin/kafka-run-class.sh kafka.tools.MirrorMaker
--consumer.config /etc/kafka/consumer-azure-event-hub.properties
--producer.config /etc/kafka/producer-azure-event-hub.properties --whitelist
"OrderPlacedEvent,OrderReturnedEvent,OrderCancelledEvent,LogisticOrderReturn,PointsChangedEvent,PointsChangedEventIT,PointsChangedEventNL,POSReceiptCreatedEvent_20210304"
EnvironmentFile=/etc/sysconfig/kafka-azure-event-hub-mm
SuccessExitStatus=143
LimitNOFILE=500000
LimitNPROC=500000
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
supposed to mirror the whitelisted topics, but as of KafkaManager only
OrderCancelledEvent
OrderPlacedEvent
LogisticOrderReturn
PointsChangedEventIT
POSReceiptCreatedEvent_20210304
show up reviewing the consumer. Checking with
kafka-consumer-groups --bootstrap-server some-kafka-node:9092 --describe
--group prd-mm-azure-event-hub-zey
it shows that
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
CONSUMER-ID HOST
CLIENT-ID
OrderPlacedEvent 0 836583 836583 0
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
PointsChangedEvent 0 - 1508747 -
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
OrderCancelledEvent 0 12410 12410 0
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
LogisticOrderReturn 0 35537 35537 0
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
OrderReturnedEvent 0 - 0 -
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
PointsChangedEventIT 0 214602 214602 0
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
POSReceiptCreatedEvent_20210304 0 66729 66729 0
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
PointsChangedEventNL 0 - 263448 -
prd-mm-azure-event-hub-zey-0-9da2d528-44c3-45bf-a9a2-428c3a9eae61/a.b.c.d
prd-mm-azure-event-hub-zey-0
...
e.g. the PointsChangedEventNL topic has no current offset but OrderPlacedEvent
has and I don’t have a clue why.
Any hint or advice would be highly appreciated (please be gentle, it’s my first
time here)
Thank you all so much, Manfred
