I don't think that is the case. The lag is huge ~10^5 records. On Thu, Jun 27, 2019 at 9:13 AM Srinath C <srinat...@gmail.com> wrote:
> Ok Garvit I still don't see the image but based on these inputs you > provided I'm thinking that the possible scenario could be that between two > polls from the consumer: > (a) the number of records added to the partitions already consumed in > previous poll is 500 or more (max.poll.records) > or > (b) the size of records in the partitions already consumed in previous poll > is around ~50Mb (fetch.max.bytes) > > Regards, > Srinath. > > > On Thu, Jun 27, 2019 at 7:41 AM Garvit Sharma <eng.gar...@gmail.com> > wrote: > > > Hi Srinath, > > > > I have attached the image. > > > > The partitions belong to the same topic only. I have not explicitly set > > max.partition.fetch.bytes or fetch.max.bytes or max.poll.records so it > > should take the default values. > > > > Let me know. > > > > Thanks, > > > > On Thu, Jun 27, 2019 at 7:11 AM Srinath C <srinat...@gmail.com> wrote: > > > >> Hi Garvit, > >> > >> Am unable to see the image you attached for some reason and am not able > to > >> see if the partitions are in the same topic or in different topics. > >> Check if any of max.partition.fetch.bytes or fetch.max.bytes or > >> max.poll.records configured in your consumer is causing the behaviour. > >> > >> Regards, > >> Srinath > >> > >> > >> On Thu, Jun 27, 2019 at 5:43 AM Peter Bukowinski <pmb...@gmail.com> > >> wrote: > >> > >> > Is there a correlation between the lagging partitions and the consumer > >> > assigned to them? > >> > > >> > > On Jun 26, 2019, at 4:25 PM, Garvit Sharma <eng.gar...@gmail.com> > >> wrote: > >> > > > >> > > Can anyone please help me with this. > >> > > > >> > > On Wed, Jun 26, 2019 at 8:56 PM Garvit Sharma <eng.gar...@gmail.com > > > >> > wrote: > >> > > > >> > >> Hey Steve, > >> > >> > >> > >> I have checked, count of messages on all the partitions are same. > >> > >> > >> > >> I am still exploring an approach using which the root cause could > be > >> > >> determined. > >> > >> > >> > >> Thanks, > >> > >> > >> > >> On Wed, Jun 26, 2019 at 8:07 PM Garvit Sharma < > eng.gar...@gmail.com> > >> > >> wrote: > >> > >> > >> > >>> I am not sure about that. Is there a way to analyse that ? > >> > >>> > >> > >>> On Wed, Jun 26, 2019 at 7:35 PM Steve Howard < > >> > steve.how...@confluent.io> > >> > >>> wrote: > >> > >>> > >> > >>>> Hi Garvit, > >> > >>>> > >> > >>>> Are the slow partitions "hot", i.e., receiving a lot more > messages > >> > than > >> > >>>> others? > >> > >>>> > >> > >>>> Thanks, > >> > >>>> > >> > >>>> Steve > >> > >>>> > >> > >>>> On Wed, Jun 26, 2019, 9:56 AM Garvit Sharma < > eng.gar...@gmail.com > >> > wrote: > >> > >>>> > >> > >>>>> Just to add more details, these consumers are processing the > Kafka > >> > >>>> events > >> > >>>>> and writing to DB(fast write guaranteed). > >> > >>>>> > >> > >>>>> On Wed, Jun 26, 2019 at 7:23 PM Garvit Sharma < > >> eng.gar...@gmail.com> > >> > >>>>> wrote: > >> > >>>>> > >> > >>>>>> Hi All, > >> > >>>>>> > >> > >>>>>> I can see huge consumer lag in a few partitions of Kafka > topic. I > >> > >>>> need to > >> > >>>>>> know the root cause of this issue. > >> > >>>>>> > >> > >>>>>> Please let me know, how to proceed. > >> > >>>>>> > >> > >>>>>> Below is sample consumer lag data : > >> > >>>>>> > >> > >>>>>> [image: image.png] > >> > >>>>>> > >> > >>>>>> Thanks, > >> > >>>>>> > >> > >>>>>> > >> > >>>>> > >> > >>>> > >> > >>> > >> > > >> > > >> > > >
