Hi Alberto, Usually this means that the leader election/replica syncing couldn't be successful and the zookeeper logs should be able to show this information too. The leader -1 is what worries me. For your case (3 broker cluster), I am assuming you have done the cluster configuration to have 1 broker-zookeeper setup ? If that's the case, you should be able to bounce 1 zookeeper at a time and see if that resolves the issue.
That said, have you restarted your servers since this issue surfaced? On 1 June 2017 at 14:11, Del Barrio, Alberto < alberto.delbar...@360dialog.com> wrote: > Hi all, > > I'm experiencing an issue which I don't know how to solve, so I'm trying to > find some guidance on the topic. > > I have a cluster composed by 3 servers, one broker per server running Kafka > 0.10.0.1-1 which runs in production with around 100 topics, most of them > divided in several partitions and replicated always between 2 servers. > Suddenly I've notice when looking at my topics (with kafka-topics tool) > that no one of them have a leader (Leader: -1) and the list of ISR appears > empty for all the topics. > So they look something like: > > Topic:mytopic PartitionCount:3 ReplicationFactor:2 Configs: > retention.ms=86400000 > Topic: mytopic Partition: 0 Leader: -1 Replicas: 30,10 Isr: > Topic: mytopic Partition: 1 Leader: -1 Replicas: 10,20 Isr: > Topic: mytopic Partition: 2 Leader: -1 Replicas: 20,30 Isr: > > However the applications using it are running normally, consumers as well > as producers. > The logs are not showing errors or weird messages with the exceptions of > some > Failed to rename [/var/log/kafka/log-cleaner.log] to > [/var/log/kafka/log-cleaner.log.2017-05-31-17] > which appear each few days. > > Now I would like to bring back to cluster to a good state. I'm afraid of > restarting brokers because all of them are supposed to be leaders for some > partitions, so if I restart them and there's no leader I might experience > data loss. > > Have you face any similar situation? Can someone give me a hint? > > Thanks in advance, > Alberto. >
