Bhavesh, unfortunately, ps cmd in Mac doesn't display thread id. I tried DTrace, but it only shows kernel thread id (not Java thread id).
anyway, I updated the jira with producer metrics. it clearly shows request rate shoot up to 18K/sec. Thanks, Steven On Wed, Feb 4, 2015 at 9:48 AM, Steven Wu <stevenz...@gmail.com> wrote: > Bhavesh, > > this is on Mac OS. I couldn't get similar options to make ps/jstack work > on Mac. will continue to try if I can make them work. > > logging output does show "kafka-producer-network-thread" sends two > metadata requests per milli-seconds. > > Thanks, > Steven > > > On Wed, Feb 4, 2015 at 9:15 AM, Bhavesh Mistry <mistry.p.bhav...@gmail.com > > wrote: > >> Hi Steven, >> >> Can you please try to see if io thread is indeed a problem ? The >> following >> on works on Linux: >> >> ps -p "$java_pid" -L -o tid,pcpu >> jstack -F "$java_pid" >> >> Then compare the thread # (may have to Hex # to decimal) between the >> Jstack >> and ps command. This will tell you which thread is consuming more CPU >> for >> that process. >> >> Thanks, >> >> Bhavesh >> >> On Wed, Feb 4, 2015 at 9:01 AM, Steven Wu <stevenz...@gmail.com> wrote: >> >> > I have re-run my unit test with 0.8.2.0. same tight-loop problem >> happened >> > after a few mins. >> > >> > On Tue, Feb 3, 2015 at 10:00 PM, Guozhang Wang <wangg...@gmail.com> >> wrote: >> > >> > > Steven, you may be hitting on KAFKA-1642 >> > > <https://issues.apache.org/jira/browse/KAFKA-1642>. >> > > >> > > As Jay said, a bunch of such issues are fixed in the new release. >> Please >> > > let us know if you still see the issue with it. >> > > >> > > Guozhang >> > > >> > > On Tue, Feb 3, 2015 at 8:52 PM, Steven Wu <stevenz...@gmail.com> >> wrote: >> > > >> > > > sure. will try my unit test again with 0.8.2.0 release tomorrow and >> > > report >> > > > back my findings. >> > > > >> > > > On Tue, Feb 3, 2015 at 8:42 PM, Jay Kreps <jay.kr...@gmail.com> >> wrote: >> > > > >> > > > > Hey Steven, >> > > > > >> > > > > That sounds like a bug. I think we fixed a few producer high cpu >> > issues >> > > > > since the beta, I wonder if you could repeat the same test with >> the >> > > > 0.8.2. >> > > > > final release? >> > > > > >> > > > > -Jay >> > > > > >> > > > > On Tue, Feb 3, 2015 at 8:37 PM, Steven Wu <stevenz...@gmail.com> >> > > wrote: >> > > > > >> > > > > > actually, my local test can reproduce the issue although not >> > > > immediately. >> > > > > > seems to happen after a few mins. I enabled TRACE level logging. >> > here >> > > > > seems >> > > > > > to be the tight loop. you can see that there are two metadata >> > > requests >> > > > in >> > > > > > one milli-seconds. >> > > > > > >> > > > > > kafka-producer-network-thread | foo 20:34:32,626 TRACE >> > > > NetworkClient:301 >> > > > > - >> > > > > > Ignoring empty metadata response with correlation id 360185. >> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG >> > > > NetworkClient:369 >> > > > > - >> > > > > > Trying to send metadata request to node -2 >> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG >> > > > NetworkClient:374 >> > > > > - >> > > > > > Sending metadata request ClientRequest(expectResponse=true, >> > > > payload=null, >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> request=RequestSend(header={api_key=3,api_version=0,correlation_id=360186,client_id=foo}, >> > > > > > body={topics=[]})) to node -2 >> > > > > > kafka-producer-network-thread | foo 20:34:32,626 TRACE >> > > > NetworkClient:301 >> > > > > - >> > > > > > Ignoring empty metadata response with correlation id 360186. >> > > > > > kafka-producer-network-thread | foo 20:34:32,626 DEBUG >> > > > NetworkClient:369 >> > > > > - >> > > > > > Trying to send metadata request to node -2 >> > > > > > >> > > > > > >> > > > > > On Tue, Feb 3, 2015 at 8:10 PM, Steven Wu <stevenz...@gmail.com >> > >> > > > wrote: >> > > > > > >> > > > > > > Hi, >> > > > > > > >> > > > > > > We have observed high cpu and high network traffic problem >> when >> > > > > > > 1) cluster (0.8.1.1) has no topic >> > > > > > > 2) KafkaProducer (0.8.2-beta) object is created without >> sending >> > any >> > > > > > traffic >> > > > > > > >> > > > > > > We have observed such problem twice. In both cases, problem >> went >> > > away >> > > > > > > immediately after one/any topic is created. >> > > > > > > >> > > > > > > Is this a known issue? Just want to check with the community >> > first >> > > > > before >> > > > > > > I spend much time to reproduce it. >> > > > > > > >> > > > > > > I couldn't reproduce the issue with similar setup with unit >> test >> > > code >> > > > > in >> > > > > > > IDE. start two brokers with no topic locally on my laptop. >> > create a >> > > > > > > KafkaProducer object without sending any msgs. but I only >> tested >> > > with >> > > > > > > 0.8.2-beta for both broker and producer. >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Steven >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > -- Guozhang >> > > >> > >> > >
