Generally, Kafka will crash when the disk fills up (it gets an exception trying to do the write to the disk). That will indeed lead to the behaviour you've talked about where all the brokers end up crashing, and there's no contingency for it in the codebase right now (and generally writing code for such scenarios is very tricky). The real trick is to monitor and ensure you don't run out of disk space at all.
On Thu, Apr 21, 2016 at 3:11 PM, Jens Rantil <jens.ran...@tink.se> wrote: > Hi Lawrence, > > > As I'm thinking through this a little more, if that is the case and the > node is removed, some partitions in the system may be marked as > under-replicated and cause a cascading effect where partitions are > re-replicated and cause other nodes to fill up. Has that ever happened? > Does Kafka have a contingency plan for such a scenario? > > Currently, Kafka doesn't rebalance partitions automagically if there is an > issue with a broker. That excludes the failure scenario that you portrait. > > Regarding how Kafka handles a full disk, I can't answer that. > > Cheers, > Jens > > On Wed, Apr 20, 2016 at 7:09 PM Lawrence Weikum <lwei...@pandora.com> > wrote: > > > Hello, > > > > I'm curious about the expected or default behavior that might occur if a > > broker in the system has filled up. By that I mean when a broker has > used > > all of its memory and disk space. Is the node simply removed from the > > system until space is cleared? > > > > As I'm thinking through this a little more, if that is the case and the > > node is removed, some partitions in the system may be marked as > > under-replicated and cause a cascading effect where partitions are > > re-replicated and cause other nodes to fill up. Has that ever happened? > > Does Kafka have a contingency plan for such a scenario? > > > > Thank you so much for your insight and all of your hard work! > > > > Lawrence > > > -- > > Jens Rantil > Backend Developer @ Tink > > Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden > For urgent matters you can reach me at +46-708-84 18 32. >
