[
https://issues.apache.org/jira/browse/KAFKA-15564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hemanth Savasere updated KAFKA-15564:
-------------------------------------
Description:
Issue : Mirror Maker 2 not replicating the proper consumer group offsets when
replication happens from source to destination kafka brokers
Steps to Reproduce :
# Created 2 Kafka clusters locally using the attached docker-compose.yml file.
Was using the confluent platform 7.2.1 docker images
# Had added the below section in docker file to create 20 topics and produce
randomly between 1000 to 2000 messages in each topic, and then consume the same
messages using consumer-groups.
{code:java}
/etc/confluent/docker/run &
sleep 20
for i in {1..20}; do
kafka-topics --create --bootstrap-server kafka-1:9092
--replication-factor 1 --partitions 1 --topic topic$$i
done
for i in {1..20}; do
num_msgs=$$(shuf -i 1000-2000 -n 1)
seq 1 $$num_msgs | kafka-console-producer --broker-list kafka-1:9092
--topic topic$$i
done
for i in {1..20}; do
timeout 10 kafka-console-consumer --bootstrap-server kafka-1:9092
--topic topic$$i --group consumer-group$$i
done
wait
{code}
# Ran the Mirror Maker 2 using the connect-mirror-maker.sh script after
downloading the Kafka 3.5.1 release binaries. Verified the 3.5.1 version was
running and commitID 2c6fb6c54472e90a was shown in attached MM2 logs file.
NOTE : Have attached the Kafka Docker file, the mirror maker 2 logs and mirror
maker 2 properties file. Also, have attached the describe command output of all
the consumer groups present in source and destination.
was:
Issue : Mirror Maker 2 not replicating the proper consumer group offsets when
replication happens from source to destination kafka brokers
Steps to Reproduce :
# Created 2 Kafka clusters locally using the attached docker-compose.yml file.
Was using the confluent platform 7.2.1 docker images
# Had added the below section in docker file to create 20 topics produce
randomly 1000 to 2000 messages in each topic, and then consume the same
messages using consumer-groups.
{code:java}
/etc/confluent/docker/run &
sleep 20
for i in {1..20}; do
kafka-topics --create --bootstrap-server kafka-1:9092
--replication-factor 1 --partitions 1 --topic topic$$i
done
for i in {1..20}; do
num_msgs=$$(shuf -i 1000-2000 -n 1)
seq 1 $$num_msgs | kafka-console-producer --broker-list kafka-1:9092
--topic topic$$i
done
for i in {1..20}; do
timeout 10 kafka-console-consumer --bootstrap-server kafka-1:9092
--topic topic$$i --group consumer-group$$i
done
wait
{code}
# Ran the Mirror Maker 2 using the connect-mirror-maker.sh script after
downloading the Kafka 3.5.1 release binaries. Verified the 3.5.1 version was
running and commitID 2c6fb6c54472e90a was shown in attached MM2 logs file.
NOTE : Have attached the Kafka Docker file, the mirror maker 2 logs and mirror
maker 2 properties file. Also, have attached the describe command output of all
the consumer groups present in source and destination.
> Kafka 3.5.1- Mirror Maker 2 replicating the wrong consumer group offsets in
> destination
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-15564
> URL: https://issues.apache.org/jira/browse/KAFKA-15564
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.5.1
> Reporter: Hemanth Savasere
> Priority: Major
> Attachments: Destination Consumer Group Offsets Describe.txt, MM2
> 3.5.1 Logs.txt, Source Consumer Group Offsets Describe.txt,
> docker-compose.yml, mm2.properties
>
>
> Issue : Mirror Maker 2 not replicating the proper consumer group offsets when
> replication happens from source to destination kafka brokers
> Steps to Reproduce :
> # Created 2 Kafka clusters locally using the attached docker-compose.yml
> file. Was using the confluent platform 7.2.1 docker images
> # Had added the below section in docker file to create 20 topics and produce
> randomly between 1000 to 2000 messages in each topic, and then consume the
> same messages using consumer-groups.
> {code:java}
> /etc/confluent/docker/run &
> sleep 20
> for i in {1..20}; do
> kafka-topics --create --bootstrap-server kafka-1:9092
> --replication-factor 1 --partitions 1 --topic topic$$i
> done
> for i in {1..20}; do
> num_msgs=$$(shuf -i 1000-2000 -n 1)
> seq 1 $$num_msgs | kafka-console-producer --broker-list
> kafka-1:9092 --topic topic$$i
> done
> for i in {1..20}; do
> timeout 10 kafka-console-consumer --bootstrap-server kafka-1:9092
> --topic topic$$i --group consumer-group$$i
> done
> wait
> {code}
> # Ran the Mirror Maker 2 using the connect-mirror-maker.sh script after
> downloading the Kafka 3.5.1 release binaries. Verified the 3.5.1 version was
> running and commitID 2c6fb6c54472e90a was shown in attached MM2 logs file.
> NOTE : Have attached the Kafka Docker file, the mirror maker 2 logs and
> mirror maker 2 properties file. Also, have attached the describe command
> output of all the consumer groups present in source and destination.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
