On 21/04/2016 15:16, Tom Crayford wrote:
Generally, Kafka will crash when the disk fills up (it gets an exception
trying to do the write to the disk). That will indeed lead to the behaviour
you've talked about where all the brokers end up crashing, and there's no
contingency for it in the codebase right now (and generally writing code
for such scenarios is very tricky). The real trick is to monitor and ensure
you don't run out of disk space at all.
We've been working on a project using Kafka and the ELK stack and run
into exactly this problem - although not with Kafka but with
Elasticsearch. Hope the following is of interest! I'm not sure how you
might apply the same to Kafka myself but perhaps there's a way...
http://www.flax.co.uk/blog/2016/04/21/running-disk-space-elasticsearch-solr/
Cheers
Charlie
On Thu, Apr 21, 2016 at 3:11 PM, Jens Rantil <jens.ran...@tink.se> wrote:
Hi Lawrence,
As I'm thinking through this a little more, if that is the case and the
node is removed, some partitions in the system may be marked as
under-replicated and cause a cascading effect where partitions are
re-replicated and cause other nodes to fill up. Has that ever happened?
Does Kafka have a contingency plan for such a scenario?
Currently, Kafka doesn't rebalance partitions automagically if there is an
issue with a broker. That excludes the failure scenario that you portrait.
Regarding how Kafka handles a full disk, I can't answer that.
Cheers,
Jens
On Wed, Apr 20, 2016 at 7:09 PM Lawrence Weikum <lwei...@pandora.com>
wrote:
Hello,
I'm curious about the expected or default behavior that might occur if a
broker in the system has filled up. By that I mean when a broker has
used
all of its memory and disk space. Is the node simply removed from the
system until space is cleared?
As I'm thinking through this a little more, if that is the case and the
node is removed, some partitions in the system may be marked as
under-replicated and cause a cascading effect where partitions are
re-replicated and cause other nodes to fill up. Has that ever happened?
Does Kafka have a contingency plan for such a scenario?
Thank you so much for your insight and all of your hard work!
Lawrence
--
Jens Rantil
Backend Developer @ Tink
Tink AB, Wallingatan 5, 111 60 Stockholm, Sweden
For urgent matters you can reach me at +46-708-84 18 32.
--
Charlie Hull
Flax - Open Source Enterprise Search
tel/fax: +44 (0)8700 118334
mobile: +44 (0)7767 825828
web: www.flax.co.uk
