Sent it too early.. We have around 5K topics and 75K partitions, with a replication factor of 3. One of our brokers went down due to disk space issues for logs. After fixing that, the broker would not come up and throw Out Of Memory issues while reading index files. It turned out that we had to tune the vm.max_map_count (the default is 65K mmap files) setting on the OS using the systctl command. Index files are mmapped and read when a broker is recovering from a crash or is restarting. I think its useful to document this on the Kafka Operations page.
sysctl -w vm.max_map_count=131070 Hope this helps someone in the same situation! Be aware of the downsides of increasing this number though. - neelesh On Wed, May 6, 2015 at 10:07 AM, Neelesh <neele...@gmail.com> wrote: > We had a situation where the kafka 0.8.2 broker would not come up >