Re: Too much log for kafka.common.KafkaException

2014-10-18 Thread xingcan
Ewen, Thanks for your prompt reply. I've read the issue. However, I think it seems to be another problem. During the logs' growing time, all consumers were kept down. And all nodes (including brokers and consumers) are disconnected due to the network problem. Besides, our new born cluster got no

Re: Too much log for kafka.common.KafkaException

2014-10-18 Thread Ewen Cheslack-Postava
This looks very similar to the error and stacktrace I see when reproducing https://issues.apache.org/jira/browse/KAFKA-1196 -- that's an overflow where the data returned in a FetchResponse exceeds 2GB. (It triggers the error you're seeing because FetchResponse's size overflows to become negative, w

Too much log for kafka.common.KafkaException

2014-10-18 Thread xingcan
Hi, all Recently, I upgrade my Kafka cluster to 0.8.1.1 and set replication with num.replica.fetchers=5. Last night, there's something wrong with the network. Soon, I found the server.log files (not data log!) on every node reached 4GB in an hour. I am not sure if it's my inappropriate configurat