I don't know the answer to the second question, if you don't use (much)
auto-generated id's for the consumer group you should be ok, since it's a
compacted topic after all, you might want to check if the compaction is on.
We set the offsets.retention.minutes to a week without a problem.

On Tue, Aug 23, 2016 at 12:21 PM Michael Freeman <mikfree...@gmail.com>
wrote:

> Might be easier to handle duplicate messages as opposed to handling long
> periods of time without messages.
>
> Michael
>
> > On 22 Aug 2016, at 15:55, Misra, Rahul <rahul.mi...@altisource.com>
> wrote:
> >
> > Hi,
> >
> > Can anybody provide any guidance on the following:
> >
> > 1. Given a limited set of groups and consumers, will increasing
> 'offsets.retention.minutes' to a high value (say 30 days) cause the
> __consumer_offsets topic to bloat unnecessarily or will compaction ensure
> that the entries for each key remain limited (which would mean that having
> a high 'offsets.retention.minutes' value is not a problem. I would prefer
> this option).
> >
> > 2. If the consumer calls commitSync() with latest already committed
> offsets (which have been committed already but no messages have been
> received for a long time after that), will it make an entry to the
> __consumer_offsets topic and ensure that the offsets are retained even with
> a small 'offsets.retention.minutes'? In our application the dry period
> (period without a new message is not well defined in advance).
> >
> >
> > Regards,
> > Rahul Misra
> >
> >
> > -----Original Message-----
> > From: Misra, Rahul [mailto:rahul.mi...@altisource.com]
> > Sent: Sunday, August 21, 2016 12:46 AM
> > To: Ian Wrigley; users@kafka.apache.org
> > Subject: RE: Offsets getting lost if no messages sent for a long time
> >
> > Hi Ian,
> >
> > Thanks for the quick response. Your answer clears things up.
> > I have some follow up questions though:
> >
> > 1. Given a limited set of groups and consumers, will increasing
> 'offsets.retention.minutes' to a high value (say 30 days) cause the
> __consumer_offsets topic to bloat unnecessarily or will compaction ensure
> that the entries for each key remain limited (which would mean that having
> a high 'offsets.retention.minutes' value is not a problem. I would prefer
> this option).
> >
> > 2. If the consumer calls commitSync() with latest already committed
> offsets (which have been committed already but no messages have been
> received for a long time after that), will it make an entry to the
> __consumer_offsets topic and ensure that the offsets are retained even with
> a small 'offsets.retention.minutes'? In our application the dry period
> (period without a new message is not well defined in advance).
> >
> >
> > Regards,
> > Rahul Misra
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Ian Wrigley [mailto:i...@confluent.io]
> > Sent: Sunday, August 21, 2016 12:01 AM
> > To: users@kafka.apache.org
> > Subject: Re: Offsets getting lost if no messages sent for a long time
> >
> > Since nothing was written to the __consumer_offsets topic for more than
> its configured retention period (offsets.retention.minutes, by default 1440
> minutes, or one day), the offset info will be removed. Retention period is
> all about when the last offset was written, not the last time a Consumer
> looked at a topic.
> >
> > You can increase the value of offsets.retention.minutes to ensure that
> offset info isn’t cleaned out before more messages are written to a topic
> and read by the Consumer (and hence the Consumer updates its offset info in
> __consumer_offsets).
> >
> > Ian.
> >
> > ---
> > Ian Wrigley
> > Director, Education Services
> > Confluent, Inc
> >
> >> On Aug 20, 2016, at 11:36 AM, Misra, Rahul <rahul.mi...@altisource.com>
> wrote:
> >>
> >> Hi,
> >>
> >> I have observed the following scenario (the consumer here has
> 'enable.auto.commit=false' and offsets are committed using commitSync() if
> any messages are received):
> >>
> >> 1.      Start a consumer (with a specific group.Id) and send some
> messages to its subscribed topic.
> >>
> >> 2.      The consumer consumes the messages and the group+consumer has
> an entry in the __commit_offsets with the latest offsets for this group and
> consumer.
> >>
> >> 3.      The consumer will keep polling the topic but don't send any
> more messages to the topic for a long time (longer than one day. The
> consumer keeps polling the topic in a while loop). The default duration for
> which a group's entry is retained in the offsets topic is 1 day.
> >>
> >> 4.      Now stop the consumer. (there is no other consumer for this
> group)
> >>
> >> 5.      Send some more messages to the topic.
> >>
> >> 6.      Start the consumer (with the same group and consumer id as
> earlier).
> >>
> >> 7.      The consumer does not pick up the new messages sent in step 5
> as it has lost the committed offsets and starts with the 'latest' offsets.
> >>
> >> Is this an expected behavior? Or do I have something wrong in the
> configurations?
> >> Is there a way to ensure that the offsets are retained even if there
> are no messages flowing in?
> >> I had  assumed that if the consumer keeps polling the kafka topic, its
> offsets will be retained even if no messages are received by the
> corresponding topic.
> >>
> >> Regards,
> >> Rahul
> >> This email message and any attachments are intended solely for the use
> of the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system. This message and any attachments may
> contain information that is confidential, privileged or exempt from
> disclosure. Delivery of this message to any person other than the intended
> recipient is not intended to waive any right or privilege. Message
> transmission is not guaranteed to be secure or free of software viruses.
> >>
> ***********************************************************************************************************************
> >
> > This email message and any attachments are intended solely for the use
> of the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system. This message and any attachments may
> contain information that is confidential, privileged or exempt from
> disclosure. Delivery of this message to any person other than the intended
> recipient is not intended to waive any right or privilege. Message
> transmission is not guaranteed to be secure or free of software viruses.
> >
> ***********************************************************************************************************************
> > This email message and any attachments are intended solely for the use
> of the addressee. If you are not the intended recipient, you are prohibited
> from reading, disclosing, reproducing, distributing, disseminating or
> otherwise using this transmission. If you have received this message in
> error, please promptly notify the sender by reply email and immediately
> delete this message from your system. This message and any attachments may
> contain information that is confidential, privileged or exempt from
> disclosure. Delivery of this message to any person other than the intended
> recipient is not intended to waive any right or privilege. Message
> transmission is not guaranteed to be secure or free of software viruses.
> >
> ***********************************************************************************************************************
>

Reply via email to