The topic created in the quickstart only has a replication factor of 1 and a single partition.
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test So the one topic partition is on broker 1. When broker 1 is up, you can consume the data but when it’s down, there is not a replica on any of the other brokers. Try creating a new topic with replication-factor 3 and you should get better availability in the event of one or even two broker failures. -hans -- /** * Hans Jespersen, Principal Systems Engineer, Confluent Inc. * h...@confluent.io (650)924-2670 */ > On Jan 19, 2017, at 9:24 PM, Steven Peh <steven....@macquarie.com> wrote: > > I’ve posted this on Stackoverflow as well: > http://stackoverflow.com/questions/41756435/kafka-0-10-quickstart-consumer-fails-when-primary-broker-is-brought-down > > > > So I'm trying the kafka quickstart<https://kafka.apache.org/quickstart> as > per the main documentation. Got the multi-cluster example all setup and test > per the instructions and it works. For example, bringing down one broker and > the producer and consumer can still send and receive. > > > > However, as per the example, we setup 3 brokers and we bring down broker 2 > (with broker id = 1). Now if I bring up all brokers again, but I bring down > broker 1 (with broker id = 0), the consumer just hangs. This only happens > with broker 1 (id = 0), does not happen with broker 2 or 3. I'm testing this > on Windows 7. > > Is there something special here with broker 1? Looking at the config they are > exactly the same between all 3 brokers except the id, port number and log > file location. > > I thought it is just a problem with the provided console consumer which > doesn't take a broker list, so I wrote a simple java consumer as per their > documentation using the default setup but specify the list of brokers in the > "bootstrap.servers" property, but no dice, still get the same problem. > > The moment I startup broker 1 (broker id = 0), the consumers will just resume > working. This isn't a highly available/fault tolerant behavior for the > consumer... any help on how to setup a HA/fault tolerant consumer? > > Producers doesn't seem to have an issue. > > > Cheers. > Steven Peh > Technology Division | Macquarie Group Services Australia Pty Limited > 50 Martin Place, Sydney, NSW, 2000, Australia > T +61 2 8237 5156 I M +61 438 444 839 I E > steven....@macquarie.com<mailto:john.st...@macquarie.com> > www.macquarie.com<http://www.macquarie.com/> > > > This email and any attachment is confidential. If you are not the intended > recipient, please delete this message. Macquarie does not guarantee the > integrity of any emails or attachments. For important disclosures and > information about the incorporation and regulated status of Macquarie Group > entities please see: www.macquarie.com/disclosures
