Could it be that you changed the KAFKA_ADVERTISED_PORT and restarted those
brokers but didn't restart the rest (until now)?
I wouldn't be surprised if the other brokers continued to use the incorrect
advertised port.
On Mon, Jul 16, 2018 at 1:40 PM, Mich Talebzadeh <mich.talebza...@gmail.com>
wrote:
> I restarted all Kafka dockers corresponding to broker 0-2 and now broker ID
> 0 is selected as leader and is working
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> --topic final
> Topic:final PartitionCount:3 ReplicationFactor:3
> Configs:
> Topic: final Partition: 0 Leader: 0 Replicas: 1,2,0
> Isr: 0
> Topic: final Partition: 1 Leader: 0 Replicas: 2,0,1
> Isr: 0
> Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
> Isr: 0
>
> That is a good one with Leader being 0 in all. But this leader selection
> was not working
>
> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> --topic final
> Topic:final PartitionCount:3 ReplicationFactor:3
> Configs:
> Topic: final Partition: 0 Leader: 1 Replicas: 1,2,0
> Isr: 1,2,0
> Topic: final Partition: 1 Leader: 2 Replicas: 2,0,1
> Isr: 2,0,1
> Topic: final Partition: 2 Leader: 0 Replicas: 0,1,2
> Isr: 0
>
> Surely something is not working here although there reports that the
> problem goes away when brokers are restarted!
>
> Thanks
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
> *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> loss, damage or destruction of data or any other property which may arise
> from relying on this email's technical content is explicitly disclaimed.
> The author will in no case be liable for any monetary damages arising from
> such loss, damage or destruction.
>
>
>
>
> On Mon, 16 Jul 2018 at 19:57, Mich Talebzadeh <mich.talebza...@gmail.com>
> wrote:
>
> > Also I noticed that bar broker ID =0, the connection to broker ID 1 )node
> > 1) and broker ID 2 (node 2) could not be established
> >
> > [2018-07-16 18:41:10,419] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,420] WARN [Producer clientId=console-producer]
> > Connection to node 2 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,464] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,470] WARN [Producer clientId=console-producer]
> > Connection to node 2 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,561] WARN [Producer clientId=console-producer]
> > Connection to node 2 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> > [2018-07-16 18:41:10,563] WARN [Producer clientId=console-producer]
> > Connection to node 1 could not be established. Broker may not be
> available.
> > (org.apache.kafka.clients.NetworkClient)
> >
> > Node 1 port 9092 is mapped to host port 9093 and node 2 port 9092 is
> > mapped to host port 9094
> >
> >
> >
> > Dr Mich Talebzadeh
> >
> >
> >
> > LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> OABUrV8Pw>*
> >
> >
> >
> > http://talebzadehmich.wordpress.com
> >
> >
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> from
> > such loss, damage or destruction.
> >
> >
> >
> >
> > On Mon, 16 Jul 2018 at 16:49, Mich Talebzadeh <mich.talebza...@gmail.com
> >
> > wrote:
> >
> >> Thanks Chris,
> >>
> >> I am afraid the issue is still there!
> >>
> >> docker run -d --name kafka_broker0 -p 9092:9092 -e
> >> KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> ZOOKEEPER_IP=50.140.197.220 -e
> >> KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 -e
> *KAFKA_ADVERTISED_PORT=9092
> >> *ches/kafka
> >>
> >> ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> >> --replication-factor 3 --partitions 3 --topic r3p3
> >>
> >> Created topic "r3p3".
> >>
> >> ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> --topic
> >> r3p3
> >>
> >> Topic:r3p3 PartitionCount:3 ReplicationFactor:3 Configs:
> >>
> >> Topic: r3p3 Partition: 0 Leader: 2 Replicas: 2,1,0
> >> Isr: 2,1,0
> >>
> >> Topic: r3p3 Partition: 1 Leader: 0 Replicas: 0,2,1
> >> Isr: 0
> >>
> >> Topic: r3p3 Partition: 2 Leader: 1 Replicas: 1,0,2
> >> Isr: 1,0,2
> >>
> >> cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> >> --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic r3p3
> >>
> >> [2018-07-16 17:01:34,496] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 10 on topic-partition r3p3-0,
> >> retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,596] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 13 on topic-partition r3p3-2,
> >> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,597] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 14 on topic-partition r3p3-0,
> >> retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,698] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 16 on topic-partition r3p3-2,
> >> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,699] WARN [Producer clientId=console-producer] Got
> >> error produce response with correlation id 17 on topic-partition r3p3-0,
> >> retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >> (org.apache.kafka.clients.producer.internals.Sender)
> >> [2018-07-16 17:01:34,800] ERROR Error when sending message to topic r3p3
> >> with key: null, value: 67 bytes with error:
> >> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> >>
> >> Dr Mich Talebzadeh
> >>
> >>
> >>
> >> LinkedIn * https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >> <https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
> >>
> >>
> >>
> >> http://talebzadehmich.wordpress.com
> >>
> >>
> >> *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >> any loss, damage or destruction of data or any other property which may
> >> arise from relying on this email's technical content is explicitly
> >> disclaimed. The author will in no case be liable for any monetary
> damages
> >> arising from such loss, damage or destruction.
> >>
> >>
> >>
> >>
> >> On Mon, 16 Jul 2018 at 16:12, Chris Richardson <
> ch...@chrisrichardson.net>
> >> wrote:
> >>
> >>> I believe you need to use -e KAFKA_ADVERTISED_PORT=909..
> >>>
> >>> On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh <
> >>> mich.talebza...@gmail.com>
> >>> wrote:
> >>>
> >>> > Hi,
> >>> >
> >>> > I have created a zookeeper and three brokers as dockers in a physical
> >>> host
> >>> > as shown below
> >>> >
> >>> > [image: image.png]
> >>> >
> >>> > The followings are used to create Zookeeper and Kafka dockers
> >>> >
> >>> > docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888
> >>> > jplock/zookeeper
> >>> >
> >>> > docker run -d --name kafka_broker0 -p 9092:9092 -e
> >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> >>> ZOOKEEPER_IP=50.140.197.220
> >>> > -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >>> >
> >>> > docker run -d --name kafka_broker1 -p 9093:9092 -e
> >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> >>> ZOOKEEPER_IP=50.140.197.220
> >>> > -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >>> >
> >>> > docker run -d --name kafka_broker2 -p 9094:9092 -e
> >>> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e
> >>> ZOOKEEPER_IP=50.140.197.220
> >>> > -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka
> >>> >
> >>> > Note the mappings of ports to the port on the physical host.
> >>> >
> >>> > I have created the following topic that works
> >>> >
> >>> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181
> >>> > --replication-factor 1 --partitions 1 --topic three
> >>> >
> >>> >
> >>> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181
> >>> --topic
> >>> > three
> >>> >
> >>> > Topic:three PartitionCount:1 ReplicationFactor:1
> >>> Configs:
> >>> >
> >>> > Topic: three Partition: 0 Leader: 0 Replicas: 0
> >>> Isr:
> >>> > 0
> >>> >
> >>> > *So there is only one partition and one replication factor*
> >>> >
> >>> > The following producer works fine
> >>> >
> >>> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh
> >>> > --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three
> >>> >
> >>> > However, when I define a topic as follows with --replication-factor 2
> >>> > --partitions 2
> >>> >
> >>> > hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181
> >>> > --replication-factor 2 --partitions 2 --topic newone
> >>> > <
> >>> > Created topic "newone".
> >>> > hduser@rhes564: /data6/hduser/prices/avg_prices>
> >>> ${KAFKA_HOME}/bin/kafka-topics.sh
> >>> > --describe -zookeeper rhes75:2181 --topic newone
> >>> > Topic:newone PartitionCount:2 ReplicationFactor:2
> >>> Configs:
> >>> > Topic: newone Partition: 0 Leader: 2 Replicas: 2,0
> >>> > Isr: 2,0
> >>> > Topic: newone Partition: 1 Leader: 0 Replicas: 0,1
> >>> > Isr: 0
> >>> >
> >>> > It throws errors!
> >>> >
> >>> > [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer]
> Got
> >>> > error produce response with correlation id 12 on topic-partition
> >>> newone-0,
> >>> > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >>> > (org.apache.kafka.clients.producer.internals.Sender)
> >>> > [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer]
> Got
> >>> > error produce response with correlation id 14 on topic-partition
> >>> newone-0,
> >>> > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION
> >>> > (org.apache.kafka.clients.producer.internals.Sender)
> >>> > [2018-07-16 15:51:41,056] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 68 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic
> >>> newone
> >>> > with key: null, value: 67 bytes with error:
> (org.apache.kafka.clients.
> >>> > producer.internals.ErrorLoggingCallback)
> >>> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> >>> > server is not the leader for that topic-partition.
> >>> >
> >>> > I believe these Kafka brokers have problem talking to each other and
> >>> the
> >>> > message is lost!
> >>> >
> >>> > Thanks
> >>> >
> >>> > Dr Mich Talebzadeh
> >>> >
> >>> >
> >>> >
> >>> > LinkedIn *
> >>> https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >>> > <
> >>> https://www.linkedin.com/profile/view?id=
> AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> >>> >*
> >>> >
> >>> >
> >>> >
> >>> > http://talebzadehmich.wordpress.com
> >>> >
> >>> >
> >>> > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> >>> any
> >>> > loss, damage or destruction of data or any other property which may
> >>> arise
> >>> > from relying on this email's technical content is explicitly
> >>> disclaimed.
> >>> > The author will in no case be liable for any monetary damages arising
> >>> from
> >>> > such loss, damage or destruction.
> >>> >
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Learn microservices - http://learn.microservices.io
> >>> Microservices application platform http://eventuate.io
> >>> Consulting and training http://chrisrichardson.net
> >>>
> >>
>
--
Learn microservices - http://learn.microservices.io
Microservices application platform http://eventuate.io
Consulting and training http://chrisrichardson.net
