Thanks every one. I"ll try to clean up the disk space and try again.
On Sun, Nov 23, 2014 at 8:47 AM, Jason Rosenberg wrote:
> Rajiv,
>
> So, any time a broker's disk fills up, it will shut itself down immediately
> (it will do this in response to any IO error on writing to disk).
> Unfortunatel
Rajiv,
So, any time a broker's disk fills up, it will shut itself down immediately
(it will do this in response to any IO error on writing to disk).
Unfortunately, this means that the node will not be able to do any
housecleaning before shutdown, which is an 'unclean' shutdown. This means
that wh
It might logs check your kafka logs dir (server logs) . Kafka can
produce lot of logs in a quick time make sure thats whats in play here.
-Harsha
On Sat, Nov 22, 2014, at 01:37 PM, Rajiv Kurian wrote:
> Actually see a bunch of errors. One of the brokers is out of space and
> this
> might be causin
Actually see a bunch of errors. One of the brokers is out of space and this
might be causing everything to spin out of control.
Some logs:
On *broker 1* (the one that has run out of space):
2014-11-22T21:20:42.790Z FATAL [ReplicaFetcherThread-1-13 ]
[kafka.server.ReplicaFetcherThread
Rajiv ,
which version of kafka are you using and do you see any errors
when the server goes down after sending few messages.
-Harsha
On Sat, Nov 22, 2014, at 01:05 PM, Rajiv Kurian wrote:
> The brokers also seem unavailable while this is going on. Each of these
> log messages
The brokers also seem unavailable while this is going on. Each of these
log messages takes 2-3 seconds so at about 1200 partitions it takes up
quite a bit of time. Ultimately it does recover though but sadly it goes
down soon enough after I start sending it messages.
On Sat, Nov 22, 2014 at 11:2
A 3 node kafka broker cluster went down yesterday (all nodes) and I just
noticed it this morning. When I restarted it this morning, I see a lengthy
list of messages like this:
Loading log 'mytopic-partitionNum"
Recovering unflushed segment 'some number' of in log mytopic-partitionNum.
Completed lo
