Hi Harper, Thanks for getting back to me- Well I figured it out eventually, and the clients now failover correctly-
The settings are almost all correct, I needed to change the "metadata.max.age.ms" setting It defaults to 5 minutes and I was restarting the brokers within less than one minute. Btw, I was checking that the brokers were indeed in the ISR list before shutting down the leader- but the clients didn't have the metadata Daniele -----Original Message----- From: Harper Henn <harper.h...@datto.com> Sent: 29 January 2019 16:45 To: users@kafka.apache.org Subject: Re: Client doesn't reconnect to the kafka-server after failover Hi Daniele, Is it possible that the IP addresses of your brokers change after you start your server instances again, and the producers can't connect because the new IPs are not in their configuration? That could also be an explanation for why the new broker is in the ISR list - I think the brokers learn about each other through zookeeper and a different IP wouldn't matter. Harper On Fri, Jan 25, 2019 at 6:10 AM Daniele Bosetti <d.bose...@cmcmarkets.com> wrote: > Hi, > > I am evaluating Kafka, and I am testing failover of kafka servers. > > My problem is, if a server comes back online after being turned off, > the client seems not to notice it. > > The setup is: > 3 zookeeper instances on boxes A,B,C > 2 kafka server instances on boxes A,B > 1 client running on the dev machine > I am using the java library, org.apache.kafka:kafka-clients:2.1.0 > > The client-producer sends messages every second, to a test topic with > replication factor=2 and partitionCount=1 The leader is broker-0 (on > box A) ; when I shut down that server instance, > broker-1 becomes the leader. > The client reports the disconnection from broker-0 and happily > connects to broker-1, messages are sent correctly. > > Then I turn broker-0 back on, and check that this instance is back in > the replica set using kafka-topics.sh (output is below). > But then, when I turn off broker-1 , the client does not reconnect to > broker-0; And I have errors > 2019-01-25 13:46:18,777 [kafka-producer-network-thread | producer-1] > DEBUG org.apache.kafka.clients.NetworkClient - [Producer > clientId=producer-1] Give up sending metadata request since no node is > available > 2019-01-25 13:46:18,828 [kafka-producer-network-thread | producer-1] > DEBUG org.apache.kafka.clients.NetworkClient - [Producer > clientId=producer-1] Give up sending metadata request since no node is > available > > > [..]$ bin/kafka-topics.sh --describe --zookeeper boxA:12181 --topic > test-topic-3 > Topic:test-topic-3 PartitionCount:1 ReplicationFactor:2 > Configs: > Topic: test-topic-3 Partition: 0 Leader: 1 Replicas: > 1,0 Isr: 1,0 > > > I am a bit of a loss, I am not sure if this is a configuration problem? > So my questions are, > (1) in a correct setup, the client should automatically reconnect to > the available broker after it comes back up? > (2) how can I fix/troubleshoot my setup? > (3) we don't need additional connection-listeners, or > cluster-listeners, on the client side (right?) > > > Any help is appreciated! > Thanks, > Daniele > ________________________________ > > > Spread bets and CFDs are complex instruments and come with a high risk > of losing money rapidly due to leverage. 80% of retail investor > accounts lose money when spread betting and/or trading CFDs with this > provider. You should consider whether you understand how spread bets > and CFDs work and whether you can afford to take the high risk of losing your > money. > Professional clients: Digital 100s and Countdowns carry a level of > risk to your capital as you could lose all of your investment. These > products may not be suitable for all clients therefore ensure you > understand the risks and seek independent advice. Invest only what you can > afford to lose. > > CMC Markets UK plc (173730) and CMC Spreadbet plc (170627) are > authorised and regulated by the Financial Conduct Authority in the > United Kingdom. CMC Markets UK plc and CMC Spreadbet plc are > registered in England and Wales with Company Numbers 02448409 and > 02589529 and with their registered offices at 133 Houndsditch, London, EC3A > 7BX. > > The content of this e-mail (including any attachments) is strictly > confidential and is for the sole use of the intended addressee(s). If > you are not the intended recipient of this e-mail please notify the > sender immediately and delete this e-mail from your system. Any > disclosure, copying, dissemination or use of its content (including > any attachments) is strictly prohibited. CMC Markets UK plc and CMC > Spreadbet plc reserve the right to intercept and monitor the content > of the e-mail messages to and from its systems. > > E-mails may be interfered with or may contain viruses or other defects > for which CMC Markets UK plc and CMC Spreadbet plc accept no > responsibility. It is the responsibility of the recipient to carry out > a virus check on the e-mail and any attachment(s). > > This communication is not intended as an offer or solicitation for the > purchase or sale of a financial instrument or as an official > confirmation of any transaction unless specifically presented as such. > > CMC Markets UK plc and CMC Spreadbet plc are registered in England and > Wales with Company Numbers 02448409 and 02589529 and with their > registered offices at 133 Houndsditch, London, EC3A 7BX. >
