[
https://issues.apache.org/jira/browse/KAFKA-15564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Savasere updated KAFKA-15564:
-------------------------------------
Description:
Issue : Mirror Maker 2 not replicating the proper consumer group offsets when
replication happens from source to destination kafka brokers
Steps to Reproduce :
# 1. Created 2 Kafka clusters locally using the attached docker-compose.yml
file. Was using the confluent platform 7.2.1 docker images
# 2. Had added the below section to create 20 topics produce randomly 1000 to
2000 messages in each topic, and then consume the same messages using
consumer-groups.
{code:java}
/etc/confluent/docker/run &
sleep 20
for i in {1..20}; do
kafka-topics --create --bootstrap-server kafka-1:9092
--replication-factor 1 --partitions 1 --topic topic$$i
done
for i in {1..20}; do
num_msgs=$$(shuf -i 1000-2000 -n 1)
seq 1 $$num_msgs | kafka-console-producer --broker-list kafka-1:9092
--topic topic$$i
done
for i in {1..20}; do
timeout 10 kafka-console-consumer --bootstrap-server kafka-1:9092
--topic topic$$i --group consumer-group$$i
done
wait
{code}
3. Ran the Mirror Maker 2 using the connect-mirror-maker.sh script after
downloading the Kafka 3.5.1 release binaries. Verified the 3.5.1 version was
running and commitID 2c6fb6c54472e90a was shown in attached MM2 logs file.
NOTE : Have attached the Kafka Docker file, the mirror maker 2 logs and mirror
maker 2 properties file. Also, have attached the describe command output of all
the consumer groups present in source and destination.
was:
Issue : Mirror Maker 2 not replicating the proper consumer group offsets when
replication happens from source to destination kafka brokers
Steps to Reproduce :
# 1. Created 2 Kafka clusters locally using the attached docker-compose.yml
file. Was using the confluent platform 7.2.1 docker images
# 2. Had added the below section to create 20 topics produce randomly 1000 to
2000 messages in each topic, and then consume the same messages using
consumer-groups.
{code:java}
/etc/confluent/docker/run &
sleep 20
for i in {1..20}; do
kafka-topics --create --bootstrap-server kafka-1:9092
--replication-factor 1 --partitions 1 --topic topic$$i
done
for i in {1..20}; do
num_msgs=$$(shuf -i 1000-2000 -n 1)
seq 1 $$num_msgs | kafka-console-producer --broker-list kafka-1:9092
--topic topic$$i
done
for i in {1..20}; do
timeout 10 kafka-console-consumer --bootstrap-server kafka-1:9092
--topic topic$$i --group consumer-group$$i
done
wait
{code}
3. Ran the Mirror Maker 2 using the connect-mirror-maker.sh script after
downloading the Kafka 3.5.1 release binaries. Verified the latest version was
running and commitID 2c6fb6c54472e90a was shown in attached MM2 logs file.
NOTE : Have attached the Kafka Docker file, the mirror maker 2 logs and mirror
maker 2 properties file. Also, have attached the describe command output of all
the consumer groups present in source and destination.
> Kafka 3.5.1- Mirror Maker 2 replicating the wrong consumer group offsets in
> destination
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-15564
> URL: https://issues.apache.org/jira/browse/KAFKA-15564
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.5.1
> Reporter: Hemanth Savasere
> Priority: Major
> Attachments: Destination Consumer Group Offsets Describe.txt, MM2
> 3.5.1 Logs.txt, Source Consumer Group Offsets Describe.txt,
> docker-compose.yml, mm2.properties
>
>
> Issue : Mirror Maker 2 not replicating the proper consumer group offsets when
> replication happens from source to destination kafka brokers
> Steps to Reproduce :
> # 1. Created 2 Kafka clusters locally using the attached docker-compose.yml
> file. Was using the confluent platform 7.2.1 docker images
> # 2. Had added the below section to create 20 topics produce randomly 1000 to
> 2000 messages in each topic, and then consume the same messages using
> consumer-groups.
> {code:java}
> /etc/confluent/docker/run &
> sleep 20
> for i in {1..20}; do
> kafka-topics --create --bootstrap-server kafka-1:9092
> --replication-factor 1 --partitions 1 --topic topic$$i
> done
> for i in {1..20}; do
> num_msgs=$$(shuf -i 1000-2000 -n 1)
> seq 1 $$num_msgs | kafka-console-producer --broker-list
> kafka-1:9092 --topic topic$$i
> done
> for i in {1..20}; do
> timeout 10 kafka-console-consumer --bootstrap-server kafka-1:9092
> --topic topic$$i --group consumer-group$$i
> done
> wait
> {code}
> 3. Ran the Mirror Maker 2 using the connect-mirror-maker.sh script after
> downloading the Kafka 3.5.1 release binaries. Verified the 3.5.1 version was
> running and commitID 2c6fb6c54472e90a was shown in attached MM2 logs file.
> NOTE : Have attached the Kafka Docker file, the mirror maker 2 logs and
> mirror maker 2 properties file. Also, have attached the describe command
> output of all the consumer groups present in source and destination.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
