Hi, Please find the inline message
________________________________ From: M. Manna <manme...@gmail.com> Sent: 13 February 2020 19:18 To: Chikulal C <chikula...@rcggs.com> Cc: Kafka Users <users@kafka.apache.org> Subject: Re: Kafka clustering issue Hi, On Thu, 13 Feb 2020 at 12:43, Chikulal C <chikula...@rcggs.com<mailto:chikula...@rcggs.com>> wrote: 1. Turned off node1 and node 2 (expected vs. actual) * expected: Message publish failure with following warnings ( in producer ) * Connection to node 0 could not be established. Broker may not be available. Connection to node 1 could not be established. Broker may not be available. * actual: The same 2. Turned on node 1 * expected: No warnings and should publish data to topic * actual: The same At this stage, what can you see regarding your topic partitions - who is leader/follower? after step2 Topic: topic2 PartitionCount: 2 ReplicationFactor: 2 Configs: Topic: topic2 Partition: 0 Leader: none Replicas: 0,1 Isr: 1 Topic: topic2 Partition: 1 Leader: none Replicas: 1,0 Isr: 1 after step 4 Topic: topic2 PartitionCount: 2 ReplicationFactor: 2 Configs: Topic: topic2 Partition: 0 Leader: 1 Replicas: 0,1 Isr: 1 Topic: topic2 Partition: 1 Leader: 1 Replicas: 1,0 Isr: 1 * 1. Turned off node 1 * expected: Message failure with following warnings * Connection to node 0 could not be established. Broker may not be available. Connection to node 1 could not be established. Broker may not be available. * actual: Message failure with only one type of warning ( It should warn me that both nodes are down ) * Connection to node 0 could not be established. Broker may not be available. Why do you think this is wrong? You have turned off the same node that you turned back on? Or, am I missing something here? --Here both nodes are down. So my assumption is that it should say both are down like the output of step 1 (Turned of node1 and node 2) * 1. Turned on node 2 * expected: No warnings and should publish data to topic * actual: Message failure with only one type of warning * Connection to node 0 could not be established. Broker may not be available. See my previous comments. I think you may have either misconfigured one of your tests, or have not explained the steps correctly. See my highlighted text above * Here when you turn on node 2 in step 4, I would like to have my cluster up, since one of the broker is up. But it is not happening. ________________________________ From: M. Manna <manme...@gmail.com<mailto:manme...@gmail.com>> Sent: 13 February 2020 17:55 To: Chikulal C <chikula...@rcggs.com<mailto:chikula...@rcggs.com>> Cc: Kafka Users <users@kafka.apache.org<mailto:users@kafka.apache.org>> Subject: Re: Kafka clustering issue My apologies as I misread one of the steps you mentioned in your original email. Could you kindly mention what you are seeing as per your order of failover tests? 1. Turned off node1 and node 2 (expected vs. actual) 2. Turned on node 1 (expected vs actual) 3. Turned off node 1 (expected vs actual) 4. Turned on node 2 (expected vs actual) Thanks, On Thu, 13 Feb 2020 at 12:06, Chikulal C <chikula...@rcggs.com<mailto:chikula...@rcggs.com>> wrote: Hi, I tried setting transaction.state.log.min.isr=1. But the issue still exists. I am also getting one warning after doing step 3 (with transaction.ate.log.min.isr=1) and producing some data on the topic as given below. [Producer clientId=producer-1] 2 partitions have leader brokers without a matching listener, including [topic2-0, topic2-1] But I was not facing this issue when transaction.state.log.min.isr was 2. This warning also leads to failure from the producer side to put data on the topic. Are there any other things I have to check? Thanks ________________________________ From: M. Manna <manme...@gmail.com<mailto:manme...@gmail.com>> Sent: 13 February 2020 16:35 To: Kafka Users <users@kafka.apache.org<mailto:users@kafka.apache.org>> Subject: Re: Kafka clustering issue This could be because you have set your transaction.ate.log.min.isr=2. Have you tried with setting this to 1? Also, please note that if your min.insync.replica=1, and you only have 2 nodes, you would only have a guarantee from 1 brokers to have the messages - but if the same broker fails then you may see issues. On Thu, 13 Feb 2020 at 10:40, Chikulal C <chikula...@rcggs.com.invalid> wrote: > Hi, > > I am facing an issue with the Kafka clustering setup that I have. I have a > Kafka cluster with two broker that are connected to two zookeepers. I am > posting data to a topic that have replication factor and partition two each > with a spring boot Kafka producer and consuming the same with another > spring boot app. > > I found one strange behavior when testing the cluster in the following > manner - > > 1. Turned off node1 and node 2 > 2. Turned on node 1 > 3. Turned off node 1 > 4. Turned on node 2 > > After turning on node 2 Kafka cluster got failed and I am not able to > produce data to Kafka. My consumer started throwing the message > continuously as given below. > > [Producer clientId=producer-1] Connection to node 1 (/server1-ip:9092) > could not be established. Broker may not be available. > > Issue is visible in both nodes. But if I kept both system up for a while > issue will get resolved and I can turn off any of the node without breaking > the cluster. > My broker configuration is as below. > > broker.id<http://broker.id>=0 > listeners=PLAINTEXT://server1-ip:9092 > advertised.listeners=PLAINTEXT://serever1-ip:9092 > num.network.threads=3 > num.io.threads=8 > socket.send.buffer.bytes=102400 > socket.receive.buffer.bytes=102400 > socket.request.max.bytes=104857600 > log.dirs=/home/user/kafka/data/kafka-logs > num.partitions=1 > num.recovery.threads.per.data.dir=2 > offsets.topic.replication.factor=2 > transaction.state.log.replication.factor=2 > transaction.state.log.min.isr=2 > log.retention.hours=168 > log.segment.bytes=1073741824 > log.retention.check.interval.ms<http://log.retention.check.interval.ms>=300000 > zookeeper.connect=serever1-ip:2181,serever2-ip:2181 > zookeeper.connection.timeout.ms<http://zookeeper.connection.timeout.ms>=6000 > group.initial.rebalance.delay.ms<http://group.initial.rebalance.delay.ms>=3000 > auto.leader.rebalance.enable=true > leader.imbalance.check.interval.seconds=5 > > Zookeeper configuration > > dataDir=/home/user/kafka/data > clientPort=2181 > maxClientCnxns=0 > initLimit=10 > syncLimit=5 > tickTime=2000 > server.1=server1-ip:2888:3888 > server.2=server2-ip:2888:3888 > > Is this is an expected behavior of Kafka or am I doing something wrong > with this configuration ? > > Can somebody help me with this issue .. > > Thanks in advance. > > >
