Hi Kant, After setting this configuration offsets.retention.minutes . I am in doubt about the two things 1. If I am deleting a topic will that topic offsets would also get deleted or will they present for 30 days? 2. What will happen if for some topics my log.retention.hours = 168 and offsets.retention.minutes= 1440 * 30 ?
Regards, Abhimanyu On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath < abhimanyunagr...@gmail.com> wrote: > @Kant I was going through the offset related configurations before setting > offsets.retention.minutes so came accross this configuration and thought to > ask whether this should also be tuned or not. > > > Regards, > Abhimanyu > > > > > On Mon, May 22, 2017 at 2:24 PM, kant kodali <kanth...@gmail.com> wrote: > >> @Abhimanyu Why do you think you need to set that? Did you try setting >> offsets.retention.minutes >> = 1440 * 30 and still seeing duplicates? >> >> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath < >> abhimanyunagr...@gmail.com> wrote: >> >> > Hi Girish , >> > >> > Do I need to tune this configuration offsets.retention.check.interv >> al.ms >> > also . Please let me know if I need to tune any other configuration. >> > >> > >> > Regards, >> > Abhimanyu >> > >> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girisha...@gmail.com> >> wrote: >> > >> > > Yup, exactly as Kant said. >> > > Also make sure that the retention of the offsets topic is an upper >> bound >> > > across all topics. So in this case, don't create any other topics in >> the >> > > future with retention of more than 30 days or otherwise they may have >> the >> > > same problem too. >> > > >> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" < >> abhimanyunagr...@gmail.com> >> > > wrote: >> > > >> > >> Hi Kant, >> > >> >> > >> Thanks for the suggestion. >> > >> >> > >> >> > >> Regards, >> > >> Abhimanyu >> > >> >> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com> >> > wrote: >> > >> >> > >>> @Abhimanyu You can try setting offset.retention = 30 >> (log.retention). >> > At >> > >>> most, you will have a storage overhead of 5 million msgs per day * >> 30 >> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you >> > have >> > >>> a >> > >>> TB of hard disk) >> > >>> >> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth...@gmail.com> >> > wrote: >> > >>> >> > >>> > Looking at that ticket and reading the comments it looks like one >> of >> > >>> the >> > >>> > concern is as follows. >> > >>> > >> > >>> > "offsets.retention.minutes is designed to handle the case that a >> > >>> consumer >> > >>> > group goes away forever. In that case, we don't want to store the >> > >>> offsets >> > >>> > for that group forever." >> > >>> > >> > >>> > This can simply be addressed by setting offset.retention == >> > >>> log.retention >> > >>> > by default right? In which case offset wont be stored forever even >> > when >> > >>> > consumer group goes away forever. When the consumer group goes >> away >> > >>> forever >> > >>> > the upper bound to clean up offsets would be equal to >> log.retention. >> > >>> > >> > >>> > >> > >>> > >> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth...@gmail.com> >> > >>> wrote: >> > >>> > >> > >>> >> What is your average message size and network speed? >> > >>> >> >> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < >> > >>> >> abhimanyunagr...@gmail.com> wrote: >> > >>> >> >> > >>> >>> Hi Girish, >> > >>> >>> >> > >>> >>> I did not set any value for offsets.retention.minutes so >> therefore >> > >>> what I >> > >>> >>> think is picking its default value i.e 1440 minutes so what do >> you >> > >>> think >> > >>> >>> what should I set if I am keeping my data for 30 days? >> > >>> >>> >> > >>> >>> Regards, >> > >>> >>> Abhimanyu >> > >>> >>> >> > >>> >> >> > >>> >> >> > >>> > >> > >>> >> > >> >> > >> >> > >> > >
