Re: Idle/dead producer connections on broker

2015-03-12 Thread Allen Wang
I wrote a simplified test program that creates 10 producers and sends a few messages each and then becomes idle. For both 0.8.1.1 and 0.8.2.0, the connections on brokers are gone once the producer instance is terminated. In prod environment where there are many many more producer instances, we ha

Re: Idle/dead producer connections on broker

2015-03-11 Thread Guozhang Wang
Hmm, this sounds like a serious bug. I do remember we have some ticket reporting similar issues before but I cannot find it now. Let me dig a bit deeper later. BTW, could you try out the 0.8.2 broker version and see if this is still easily re-producible, i.e. starting a bunch of producers to send

Idle/dead producer connections on broker

2015-03-10 Thread Allen Wang
Hello, We are using Kafka 0.8.1.1 on the broker and 0.8.2 producer on the client. After running for a few days, we have found that there are way too many open file descriptors on the broker side. When we compare the connections on the client side, we found some connections are already gone on the