Also I noticed that bar broker ID =0, the connection to broker ID 1 )node 1) and broker ID 2 (node 2) could not be established
[2018-07-16 18:41:10,419] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-07-16 18:41:10,420] WARN [Producer clientId=console-producer] Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-07-16 18:41:10,464] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-07-16 18:41:10,470] WARN [Producer clientId=console-producer] Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-07-16 18:41:10,561] WARN [Producer clientId=console-producer] Connection to node 2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2018-07-16 18:41:10,563] WARN [Producer clientId=console-producer] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) Node 1 port 9092 is mapped to host port 9093 and node 2 port 9092 is mapped to host port 9094 Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On Mon, 16 Jul 2018 at 16:49, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks Chris, > > I am afraid the issue is still there! > > docker run -d --name kafka_broker0 -p 9092:9092 -e > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 -e > KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 -e *KAFKA_ADVERTISED_PORT=9092 > *ches/kafka > > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181 > --replication-factor 3 --partitions 3 --topic r3p3 > > Created topic "r3p3". > > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181 --topic > r3p3 > > Topic:r3p3 PartitionCount:3 ReplicationFactor:3 Configs: > > Topic: r3p3 Partition: 0 Leader: 2 Replicas: 2,1,0 > Isr: 2,1,0 > > Topic: r3p3 Partition: 1 Leader: 0 Replicas: 0,2,1 > Isr: 0 > > Topic: r3p3 Partition: 2 Leader: 1 Replicas: 1,0,2 > Isr: 1,0,2 > > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh --broker-list > rhes75:9092, rhes75:9093, rhes75:9094 --topic r3p3 > > [2018-07-16 17:01:34,496] WARN [Producer clientId=console-producer] Got > error produce response with correlation id 10 on topic-partition r3p3-0, > retrying (2 attempts left). Error: NOT_LEADER_FOR_PARTITION > (org.apache.kafka.clients.producer.internals.Sender) > [2018-07-16 17:01:34,596] WARN [Producer clientId=console-producer] Got > error produce response with correlation id 13 on topic-partition r3p3-2, > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION > (org.apache.kafka.clients.producer.internals.Sender) > [2018-07-16 17:01:34,597] WARN [Producer clientId=console-producer] Got > error produce response with correlation id 14 on topic-partition r3p3-0, > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION > (org.apache.kafka.clients.producer.internals.Sender) > [2018-07-16 17:01:34,698] WARN [Producer clientId=console-producer] Got > error produce response with correlation id 16 on topic-partition r3p3-2, > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION > (org.apache.kafka.clients.producer.internals.Sender) > [2018-07-16 17:01:34,699] WARN [Producer clientId=console-producer] Got > error produce response with correlation id 17 on topic-partition r3p3-0, > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION > (org.apache.kafka.clients.producer.internals.Sender) > [2018-07-16 17:01:34,800] ERROR Error when sending message to topic r3p3 > with key: null, value: 67 bytes with error: > (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Mon, 16 Jul 2018 at 16:12, Chris Richardson <ch...@chrisrichardson.net> > wrote: > >> I believe you need to use -e KAFKA_ADVERTISED_PORT=909.. >> >> On Mon, Jul 16, 2018 at 7:41 AM, Mich Talebzadeh < >> mich.talebza...@gmail.com> >> wrote: >> >> > Hi, >> > >> > I have created a zookeeper and three brokers as dockers in a physical >> host >> > as shown below >> > >> > [image: image.png] >> > >> > The followings are used to create Zookeeper and Kafka dockers >> > >> > docker run -d --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 >> > jplock/zookeeper >> > >> > docker run -d --name kafka_broker0 -p 9092:9092 -e >> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 >> > -e KAFKA_BROKER_ID=0 -e KAFKA_BROKER_PORT=9092 ches/kafka >> > >> > docker run -d --name kafka_broker1 -p 9093:9092 -e >> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 >> > -e KAFKA_BROKER_ID=1 -e KAFKA_BROKER_PORT=9092 ches/kafka >> > >> > docker run -d --name kafka_broker2 -p 9094:9092 -e >> > KAFKA_ADVERTISED_HOST_NAME=50.140.197.220 -e ZOOKEEPER_IP=50.140.197.220 >> > -e KAFKA_BROKER_ID=2 -e KAFKA_BROKER_PORT=9092 ches/kafka >> > >> > Note the mappings of ports to the port on the physical host. >> > >> > I have created the following topic that works >> > >> > ${KAFKA_HOME}/bin/kafka-topics.sh --create --zookeeper rhes75:2181 >> > --replication-factor 1 --partitions 1 --topic three >> > >> > >> > ${KAFKA_HOME}/bin/kafka-topics.sh --describe -zookeeper rhes75:2181 >> --topic >> > three >> > >> > Topic:three PartitionCount:1 ReplicationFactor:1 Configs: >> > >> > Topic: three Partition: 0 Leader: 0 Replicas: 0 >> Isr: >> > 0 >> > >> > *So there is only one partition and one replication factor* >> > >> > The following producer works fine >> > >> > cat ${IN_FILE} | ${KAFKA_HOME}/bin/kafka-console-producer.sh >> > --broker-list rhes75:9092, rhes75:9093, rhes75:9094 --topic three >> > >> > However, when I define a topic as follows with --replication-factor 2 >> > --partitions 2 >> > >> > hduser@rhes564: /data6/hduser/prices/avg_prices> rhes75:2181 >> > --replication-factor 2 --partitions 2 --topic newone >> > < >> > Created topic "newone". >> > hduser@rhes564: /data6/hduser/prices/avg_prices> >> ${KAFKA_HOME}/bin/kafka-topics.sh >> > --describe -zookeeper rhes75:2181 --topic newone >> > Topic:newone PartitionCount:2 ReplicationFactor:2 Configs: >> > Topic: newone Partition: 0 Leader: 2 Replicas: 2,0 >> > Isr: 2,0 >> > Topic: newone Partition: 1 Leader: 0 Replicas: 0,1 >> > Isr: 0 >> > >> > It throws errors! >> > >> > [2018-07-16 15:51:40,852] WARN [Producer clientId=console-producer] Got >> > error produce response with correlation id 12 on topic-partition >> newone-0, >> > retrying (1 attempts left). Error: NOT_LEADER_FOR_PARTITION >> > (org.apache.kafka.clients.producer.internals.Sender) >> > [2018-07-16 15:51:40,955] WARN [Producer clientId=console-producer] Got >> > error produce response with correlation id 14 on topic-partition >> newone-0, >> > retrying (0 attempts left). Error: NOT_LEADER_FOR_PARTITION >> > (org.apache.kafka.clients.producer.internals.Sender) >> > [2018-07-16 15:51:41,056] ERROR Error when sending message to topic >> newone >> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients. >> > producer.internals.ErrorLoggingCallback) >> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This >> > server is not the leader for that topic-partition. >> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic >> newone >> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients. >> > producer.internals.ErrorLoggingCallback) >> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This >> > server is not the leader for that topic-partition. >> > [2018-07-16 15:51:41,059] ERROR Error when sending message to topic >> newone >> > with key: null, value: 68 bytes with error: (org.apache.kafka.clients. >> > producer.internals.ErrorLoggingCallback) >> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This >> > server is not the leader for that topic-partition. >> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic >> newone >> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients. >> > producer.internals.ErrorLoggingCallback) >> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This >> > server is not the leader for that topic-partition. >> > [2018-07-16 15:51:41,060] ERROR Error when sending message to topic >> newone >> > with key: null, value: 67 bytes with error: (org.apache.kafka.clients. >> > producer.internals.ErrorLoggingCallback) >> > org.apache.kafka.common.errors.NotLeaderForPartitionException: This >> > server is not the leader for that topic-partition. >> > >> > I believe these Kafka brokers have problem talking to each other and the >> > message is lost! >> > >> > Thanks >> > >> > Dr Mich Talebzadeh >> > >> > >> > >> > LinkedIn * >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > < >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> >* >> > >> > >> > >> > http://talebzadehmich.wordpress.com >> > >> > >> > *Disclaimer:* Use it at your own risk. Any and all responsibility for >> any >> > loss, damage or destruction of data or any other property which may >> arise >> > from relying on this email's technical content is explicitly disclaimed. >> > The author will in no case be liable for any monetary damages arising >> from >> > such loss, damage or destruction. >> > >> > >> > >> >> >> >> -- >> Learn microservices - http://learn.microservices.io >> Microservices application platform http://eventuate.io >> Consulting and training http://chrisrichardson.net >> >
