We have 3 DC's and created 5 node Kafka cluster in each DC, connected these 3 
DC's using Mirror Maker for replication. We were conducting performance testing 
using Kafka Producer Performance tool to load 100 million rows into 7 topics. 
We expected that data will be loaded evenly across 7 topics but 4 topics got 
loaded with ~2 million messages and remaining 3 topics loaded with 90 million 
messages. The nodes that were leaders of those 3 topics ran out of disk space 
and nodes went down. 
We tried to bring back these 2 nodes by doing following
1. Stopped Kafka Service 2. Deleted couple of topics that were taking up too 
much space i.e. /var/kafka/logs/{topic$}/ and file system showed 47% available 
3. Brought back the Kafka nodes
As soon as nodes are back, we started observing the file system growing and in 
15 minutes the mount point became full again. Deleted topics got recreated and 
taking up space again. Looking at kafka.log, it shows many of the following 
messages. Ultimately the node goes down. We don't need to recover data now, we 
would like to bring nodes back. What are the steps to bring back these nodes?
[2015-03-11 20:52:36,323] INFO Rolled new log segment for 'dc2-perf-topic5-0' 
in 3 ms. (kafka.log.Log)
[2015-03-11 15:58:07,321] INFO [Kafka Server 1021124614], started 
(kafka.server.KafkaServer)
[2015-03-11 15:58:07,882] INFO Completed load of log dc2-perf-topic5-0 with log 
end offset 0 (kafka.log.Log)
[2015-03-11 15:58:07,900] INFO Created log for partition [dc2-perf-topic5,0] in 
/var/kafka/log with properties {segment.index.bytes -> 10485760, 
file.delete.delay.ms -> 60000, segment.bytes -> 1073741824, flush.ms -> 
9223372036854775807, delete.retention.ms -> 3600000, index.interval.bytes -> 
4096, retention.bytes -> -1, cleanup.policy -> delete, segment.ms -> 604800000, 
max.message.bytes -> 1000012, flush.messages -> 9223372036854775807, 
min.cleanable.dirty.ratio -> 0.5, retention.ms -> 604800000}. 
(kafka.log.LogManager)
[2015-03-11 15:58:07,914] INFO Completed load of log dc2-perf-topic2-0 with log 
end offset 0 (kafka.log.Log)
[2015-03-11 15:58:07,916] INFO Created log for partition [dc2-perf-topic2,0] in 
/var/kafka/log with properties {segment.index.bytes -> 10485760, 
file.delete.delay.ms -> 60000, segment.bytes -> 1073741824, flush.ms -> 
9223372036854775807, delete.retention.ms -> 3600000, index.interval.bytes -> 
4096, retention.bytes -> -1, cleanup.policy -> delete, segment.ms -> 604800000, 
max.message.bytes -> 1000012, flush.messages -> 9223372036854775807, 
min.cleanable.dirty.ratio -> 0.5, retention.ms -> 604800000}. 
(kafka.log.LogManager)
[2015-03-11 15:58:07,935] INFO Completed load of log dc2-perf-topic9-0 with log 
end offset 0 (kafka.log.Log)

 SP Naidu 

Reply via email to