Avi Chopra created KAFKA-4091:
---------------------------------
Summary: Unable to produce or consume on any topic
Key: KAFKA-4091
URL: https://issues.apache.org/jira/browse/KAFKA-4091
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.10.0.0
Environment: Amazon Linux, t2.micro
Reporter: Avi Chopra
Priority: Critical
While trying to set kafka on 2 slave and 1 master box, got a weird condition
where I was not able to consume or produce to a topic.
Using Mirror Maker to sync data between slave <--> Master. Getting following
logs unending :
[2016-08-26 14:28:33,897] WARN Bootstrap broker localhost:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:43,515] WARN
Bootstrap broker localhost:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:45,118] WARN
Bootstrap broker localhost:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:46,721] WARN
Bootstrap broker localhost:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:48,324] WARN
Bootstrap broker localhost:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:49,927] WARN
Bootstrap broker localhost:9092 disconnected
(org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:53,029] WARN
Bootstrap broker localhost:9092 disconnected
(org.apache.kafka.clients.NetworkClient)
Only way I could recover was by restarting Kafka which produced this kind of
logs :
[2016-08-26 14:30:54,856] WARN Found a corrupted index file,
/tmp/kafka-logs/__consumer_offsets-43/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log) [2016-08-26 14:30:54,856] INFO Recovering
unflushed segment 0 in log __consumer_offsets-43. (kafka.log.Log) [2016-08-26
14:30:54,857] INFO Completed load of log __consumer_offsets-43 with log end
offset 0 (kafka.log.Log) [2016-08-26 14:30:54,860] WARN Found a corrupted index
file, /tmp/kafka-logs/__consumer_offsets-26/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log) [2016-08-26 14:30:54,860] INFO
Recovering unflushed segment 0 in log __consumer_offsets-26. (kafka.log.Log)
[2016-08-26 14:30:54,861] INFO Completed load of log __consumer_offsets-26 with
log end offset 0 (kafka.log.Log) [2016-08-26 14:30:54,864] WARN Found a
corrupted index file,
/tmp/kafka-logs/__consumer_offsets-35/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log)
ERROR Error when sending message to topic dr_ubr_analytics_limits with key:
null, value: 1 bytes with error:
(org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Failed to update metadata
after 60000 ms.
The consumer group command was showing a major lag.
This is my test phase so I was able to restart and recover from the master box
but I want know what caused this issue and how can it be avoided. Is there a
way to debug this issue?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
