Re: high cpu and network traffic when cluster has no topic

Sounds like this patch fixed the issue. It would be good to get some review on KAFKA-1919--it is only a four line change. On Wed, Feb 4, 2015 at 1:15 PM, Steven Wu wrote: > Bhavesh, > > unfortunately, ps cmd in Mac doesn't display thread id. I tried DTrace, but > it only shows kernel thread id (

Re: high cpu and network traffic when cluster has no topic

Bhavesh, unfortunately, ps cmd in Mac doesn't display thread id. I tried DTrace, but it only shows kernel thread id (not Java thread id). anyway, I updated the jira with producer metrics. it clearly shows request rate shoot up to 18K/sec. Thanks, Steven On Wed, Feb 4, 2015 at 9:48 AM, Steven Wu

Re: high cpu and network traffic when cluster has no topic

Bhavesh, this is on Mac OS. I couldn't get similar options to make ps/jstack work on Mac. will continue to try if I can make them work. logging output does show "kafka-producer-network-thread" sends two metadata requests per milli-seconds. Thanks, Steven On Wed, Feb 4, 2015 at 9:15 AM, Bhavesh

Re: high cpu and network traffic when cluster has no topic

Hi Steven, Can you please try to see if io thread is indeed a problem ? The following on works on Linux: ps -p "$java_pid" -L -o tid,pcpu jstack -F "$java_pid" Then compare the thread # (may have to Hex # to decimal) between the Jstack and ps command. This will tell you which thread is consu

Re: high cpu and network traffic when cluster has no topic

That's a bug then, filed https://issues.apache.org/jira/browse/KAFKA-1919 -Jay On Wed, Feb 4, 2015 at 9:01 AM, Steven Wu wrote: > I have re-run my unit test with 0.8.2.0. same tight-loop problem happened > after a few mins. > > On Tue, Feb 3, 2015 at 10:00 PM, Guozhang Wang wrote: > > > Steven

Re: high cpu and network traffic when cluster has no topic

I have re-run my unit test with 0.8.2.0. same tight-loop problem happened after a few mins. On Tue, Feb 3, 2015 at 10:00 PM, Guozhang Wang wrote: > Steven, you may be hitting on KAFKA-1642 > . > > As Jay said, a bunch of such issues are fixed in

Re: high cpu and network traffic when cluster has no topic

Steven, you may be hitting on KAFKA-1642 . As Jay said, a bunch of such issues are fixed in the new release. Please let us know if you still see the issue with it. Guozhang On Tue, Feb 3, 2015 at 8:52 PM, Steven Wu wrote: > sure. will try my un

Re: high cpu and network traffic when cluster has no topic

sure. will try my unit test again with 0.8.2.0 release tomorrow and report back my findings. On Tue, Feb 3, 2015 at 8:42 PM, Jay Kreps wrote: > Hey Steven, > > That sounds like a bug. I think we fixed a few producer high cpu issues > since the beta, I wonder if you could repeat the same test wit

Re: high cpu and network traffic when cluster has no topic

Hey Steven, That sounds like a bug. I think we fixed a few producer high cpu issues since the beta, I wonder if you could repeat the same test with the 0.8.2. final release? -Jay On Tue, Feb 3, 2015 at 8:37 PM, Steven Wu wrote: > actually, my local test can reproduce the issue although not imm

Re: high cpu and network traffic when cluster has no topic

actually, my local test can reproduce the issue although not immediately. seems to happen after a few mins. I enabled TRACE level logging. here seems to be the tight loop. you can see that there are two metadata requests in one milli-seconds. kafka-producer-network-thread | foo 20:34:32,626 TRACE