Hey, I am working on a simplified test case to check if there is any issue in my code. Just to make sure that any of my assumptions are not wrong, it will be great if you can please help me in finding answers to following queries:-
1) Is it correct to say that each commitSync will trigger a HeartBeatTask? If there is no hear beat sent in past since specified heartbeat interval then i should see a successful heartbeat response or failure message in logs near to commitSync success log? 2) is it correct to say that Meta Data refresh will not act as heartbeat, will not trigger heartBeatTask and will not reset heartBeatTask? 3) Where does a consumer session maintained? Lets say my consumer is listening to 3 partitions on a 3 broker cluster where each broker is leader of 1 partition. So will each of the brokers will have a session for my consumer or is it just 1 session maintained somewhere in common like zookeeper? 4) In above setup, during a long processing if I commit a record through commmitSync which triggers a hear beat request and a successful response is received for the same then what does this response means? does it mean that my session with each broker is renewed? or does it mean that just the leader for partition of committed record knows that my consumer is alive and consumer's session on other brokers will still timeout? Regards, Vinay Sharma On Tue, Apr 26, 2016 at 2:38 PM, Jason Gustafson <ja...@confluent.io> wrote: > Hey Vinay, > > Are you saying that heartbeats are not sent while a metadata refresh is in > progress? Do you have any logs which show us the apparent problem? > > Thanks, > Jason > > On Tue, Apr 26, 2016 at 8:18 AM, vinay sharma <vinsharma.t...@gmail.com> > wrote: > > > Hi Ismael, > > > > Treating commitSync as heartbeat will definitely resolve the issue i am > > facing but the reason behind my issue does not seem to be what mentioned > in > > defect (i.e frequent commitSync requests). > > > > I am sending CommitSync periodically only to keep my session alive when > my > > consumer is still processing records and is close to session time out > > (tried 10th / 12th / 15th / 20th second after poll called where session > > time is 30). I see heartbeat response received in logs along with each > > commitSync call but this stops after a meta data refresh request is > issued. > > I see in logs that commit goes successful but no heartbeat response > > received message in logs after meta refresh till next poll. > > > > Regards, > > Vinay Sharma > > > > On Mon, Apr 25, 2016 at 5:06 PM, Ismael Juma <ism...@juma.me.uk> wrote: > > > > > Hi Vinay, > > > > > > This was fixed via https://issues.apache.org/jira/browse/KAFKA-3470 > > (will > > > be part of 0.10.0.0). > > > > > > Ismael > > > > > > > > > > > > On Mon, Apr 25, 2016 at 1:52 PM, vinay sharma < > vinsharma.t...@gmail.com> > > > wrote: > > > > > > > Hello, > > > > > > > > I am using client API 0.9.0.1 and facing an issue. As per my logs it > > > seems > > > > that on each commitSync(Offsets) a heartbeat request is sent but > after > > a > > > > metada refresh request till next poll(), commits do not send any > > hearbeat > > > > request. > > > > > > > > KafkaConsumers i create sometimes get session time out due to no > > hearbeat > > > > specially during longer processing times. I call CommitSync(offsets) > > > after > > > > regular intervals to keep session alive when processing takes longer > > than > > > > usual. Every thing works fine if commit intervals are very small or > if > > i > > > > commit after each record but if i commit lets say every 12 seconds > and > > 30 > > > > seconds is session time then i can see consumer getting timed out > > > > sometimes. > > > > > > > > Any help or pointers will be much appreciated. Thanks in advance. > > > > > > > > Regards, > > > > Vinay sharma > > > > > > > > > >
