Anukool Rattana created KAFKA-4368:
--------------------------------------
Summary: Unclean shutdown breaks Kafka cluster
Key: KAFKA-4368
URL: https://issues.apache.org/jira/browse/KAFKA-4368
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.10.0.0, 0.9.0.1
Reporter: Anukool Rattana
Priority: Critical
My team has observed that if broker process die unclean then it will block
producer from sending messages to kafka topic.
Here is how to reproduce the problem:
1) Create a Kafka 0.10 with three brokers (A, B and C).
2) Create topic with replication_factor = 2
3) Set producer to send messages with "acks=all" meaning all replicas must be
created before able to proceed next message.
4) Force IEM (IBM Endpoint Manager) to send patch to broker A and force server
to reboot after patches installed.
Note: min.insync.replicas = 1
Result: - Producers are not able send messages to kafka topic after broker
rebooted and come back to join cluster with following error messages.
[2016-09-28 09:32:41,823] WARN Error while fetching metadata with correlation
id 0 : {logstash=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)
We suspected that number of replication_factor (2) is not sufficient to our
kafka environment but really need an explanation on what happen when broker
facing unclean shutdown.
The same issue occurred when setting cluster with 2 brokers and
replication_factor = 1.
The workaround i used to recover service is to cleanup both kafka topic log
file and zookeeper data (rmr /brokers/topics/XXX and rmr /consumers/XXX).
Note:
Topic list after A comeback from rebooted.
Topic:logstash PartitionCount:3 ReplicationFactor:2 Configs:
Topic: logstash Partition: 0 Leader: 1 Replicas: 1,3 Isr: 1,3
Topic: logstash Partition: 1 Leader: 2 Replicas: 2,1 Isr: 2,1
Topic: logstash Partition: 2 Leader: 3 Replicas: 3,2 Isr: 2,3
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
