@Abhimanyu 1) My guess is that topic offsets will remain for 30 days since that is the configuration you are explicitly setting and Kafka should respect that although I don't know for sure.
2) same as #1 offsets should remain to whatever time you specify. What is the problem with setting offsets.retention.minutes == log.retention.hours ? Did it fix the problem you were facing before? On Tue, May 23, 2017 at 11:40 PM, Abhimanyu Nagrath < abhimanyunagr...@gmail.com> wrote: > Hi Kant, > > After setting this configuration offsets.retention.minutes . I am in doubt > about the two things > 1. If I am deleting a topic will that topic offsets would also get deleted > or will they present for 30 days? > 2. What will happen if for some topics my log.retention.hours = 168 and > offsets.retention.minutes= 1440 * 30 ? > > Regards, > Abhimanyu > > On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath < > abhimanyunagr...@gmail.com> wrote: > > > @Kant I was going through the offset related configurations before > setting > > offsets.retention.minutes so came accross this configuration and thought > to > > ask whether this should also be tuned or not. > > > > > > Regards, > > Abhimanyu > > > > > > > > > > On Mon, May 22, 2017 at 2:24 PM, kant kodali <kanth...@gmail.com> wrote: > > > >> @Abhimanyu Why do you think you need to set that? Did you try setting > >> offsets.retention.minutes > >> = 1440 * 30 and still seeing duplicates? > >> > >> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath < > >> abhimanyunagr...@gmail.com> wrote: > >> > >> > Hi Girish , > >> > > >> > Do I need to tune this configuration offsets.retention.check.interv > >> al.ms > >> > also . Please let me know if I need to tune any other configuration. > >> > > >> > > >> > Regards, > >> > Abhimanyu > >> > > >> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girisha...@gmail.com> > >> wrote: > >> > > >> > > Yup, exactly as Kant said. > >> > > Also make sure that the retention of the offsets topic is an upper > >> bound > >> > > across all topics. So in this case, don't create any other topics in > >> the > >> > > future with retention of more than 30 days or otherwise they may > have > >> the > >> > > same problem too. > >> > > > >> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" < > >> abhimanyunagr...@gmail.com> > >> > > wrote: > >> > > > >> > >> Hi Kant, > >> > >> > >> > >> Thanks for the suggestion. > >> > >> > >> > >> > >> > >> Regards, > >> > >> Abhimanyu > >> > >> > >> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com> > >> > wrote: > >> > >> > >> > >>> @Abhimanyu You can try setting offset.retention = 30 > >> (log.retention). > >> > At > >> > >>> most, you will have a storage overhead of 5 million msgs per day * > >> 30 > >> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since > you > >> > have > >> > >>> a > >> > >>> TB of hard disk) > >> > >>> > >> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth...@gmail.com> > >> > wrote: > >> > >>> > >> > >>> > Looking at that ticket and reading the comments it looks like > one > >> of > >> > >>> the > >> > >>> > concern is as follows. > >> > >>> > > >> > >>> > "offsets.retention.minutes is designed to handle the case that a > >> > >>> consumer > >> > >>> > group goes away forever. In that case, we don't want to store > the > >> > >>> offsets > >> > >>> > for that group forever." > >> > >>> > > >> > >>> > This can simply be addressed by setting offset.retention == > >> > >>> log.retention > >> > >>> > by default right? In which case offset wont be stored forever > even > >> > when > >> > >>> > consumer group goes away forever. When the consumer group goes > >> away > >> > >>> forever > >> > >>> > the upper bound to clean up offsets would be equal to > >> log.retention. > >> > >>> > > >> > >>> > > >> > >>> > > >> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali < > kanth...@gmail.com> > >> > >>> wrote: > >> > >>> > > >> > >>> >> What is your average message size and network speed? > >> > >>> >> > >> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < > >> > >>> >> abhimanyunagr...@gmail.com> wrote: > >> > >>> >> > >> > >>> >>> Hi Girish, > >> > >>> >>> > >> > >>> >>> I did not set any value for offsets.retention.minutes so > >> therefore > >> > >>> what I > >> > >>> >>> think is picking its default value i.e 1440 minutes so what do > >> you > >> > >>> think > >> > >>> >>> what should I set if I am keeping my data for 30 days? > >> > >>> >>> > >> > >>> >>> Regards, > >> > >>> >>> Abhimanyu > >> > >>> >>> > >> > >>> >> > >> > >>> >> > >> > >>> > > >> > >>> > >> > >> > >> > >> > >> > > >> > > > > >
