@Kant I was going through the offset related configurations before setting offsets.retention.minutes so came accross this configuration and thought to ask whether this should also be tuned or not.
Regards, Abhimanyu On Mon, May 22, 2017 at 2:24 PM, kant kodali <kanth...@gmail.com> wrote: > @Abhimanyu Why do you think you need to set that? Did you try setting > offsets.retention.minutes > = 1440 * 30 and still seeing duplicates? > > On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath < > abhimanyunagr...@gmail.com> wrote: > > > Hi Girish , > > > > Do I need to tune this configuration offsets.retention.check.interval.ms > > also . Please let me know if I need to tune any other configuration. > > > > > > Regards, > > Abhimanyu > > > > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girisha...@gmail.com> > wrote: > > > > > Yup, exactly as Kant said. > > > Also make sure that the retention of the offsets topic is an upper > bound > > > across all topics. So in this case, don't create any other topics in > the > > > future with retention of more than 30 days or otherwise they may have > the > > > same problem too. > > > > > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <abhimanyunagr...@gmail.com > > > > > wrote: > > > > > >> Hi Kant, > > >> > > >> Thanks for the suggestion. > > >> > > >> > > >> Regards, > > >> Abhimanyu > > >> > > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com> > > wrote: > > >> > > >>> @Abhimanyu You can try setting offset.retention = 30 (log.retention). > > At > > >>> most, you will have a storage overhead of 5 million msgs per day * 30 > > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you > > have > > >>> a > > >>> TB of hard disk) > > >>> > > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth...@gmail.com> > > wrote: > > >>> > > >>> > Looking at that ticket and reading the comments it looks like one > of > > >>> the > > >>> > concern is as follows. > > >>> > > > >>> > "offsets.retention.minutes is designed to handle the case that a > > >>> consumer > > >>> > group goes away forever. In that case, we don't want to store the > > >>> offsets > > >>> > for that group forever." > > >>> > > > >>> > This can simply be addressed by setting offset.retention == > > >>> log.retention > > >>> > by default right? In which case offset wont be stored forever even > > when > > >>> > consumer group goes away forever. When the consumer group goes away > > >>> forever > > >>> > the upper bound to clean up offsets would be equal to > log.retention. > > >>> > > > >>> > > > >>> > > > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth...@gmail.com> > > >>> wrote: > > >>> > > > >>> >> What is your average message size and network speed? > > >>> >> > > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < > > >>> >> abhimanyunagr...@gmail.com> wrote: > > >>> >> > > >>> >>> Hi Girish, > > >>> >>> > > >>> >>> I did not set any value for offsets.retention.minutes so > therefore > > >>> what I > > >>> >>> think is picking its default value i.e 1440 minutes so what do > you > > >>> think > > >>> >>> what should I set if I am keeping my data for 30 days? > > >>> >>> > > >>> >>> Regards, > > >>> >>> Abhimanyu > > >>> >>> > > >>> >> > > >>> >> > > >>> > > > >>> > > >> > > >> > > >
