Hi Girish , Do I need to tune this configuration offsets.retention.check.interval.ms also . Please let me know if I need to tune any other configuration.
Regards, Abhimanyu On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girisha...@gmail.com> wrote: > Yup, exactly as Kant said. > Also make sure that the retention of the offsets topic is an upper bound > across all topics. So in this case, don't create any other topics in the > future with retention of more than 30 days or otherwise they may have the > same problem too. > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <abhimanyunagr...@gmail.com> > wrote: > >> Hi Kant, >> >> Thanks for the suggestion. >> >> >> Regards, >> Abhimanyu >> >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com> wrote: >> >>> @Abhimanyu You can try setting offset.retention = 30 (log.retention). At >>> most, you will have a storage overhead of 5 million msgs per day * 30 >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since you have >>> a >>> TB of hard disk) >>> >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth...@gmail.com> wrote: >>> >>> > Looking at that ticket and reading the comments it looks like one of >>> the >>> > concern is as follows. >>> > >>> > "offsets.retention.minutes is designed to handle the case that a >>> consumer >>> > group goes away forever. In that case, we don't want to store the >>> offsets >>> > for that group forever." >>> > >>> > This can simply be addressed by setting offset.retention == >>> log.retention >>> > by default right? In which case offset wont be stored forever even when >>> > consumer group goes away forever. When the consumer group goes away >>> forever >>> > the upper bound to clean up offsets would be equal to log.retention. >>> > >>> > >>> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth...@gmail.com> >>> wrote: >>> > >>> >> What is your average message size and network speed? >>> >> >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < >>> >> abhimanyunagr...@gmail.com> wrote: >>> >> >>> >>> Hi Girish, >>> >>> >>> >>> I did not set any value for offsets.retention.minutes so therefore >>> what I >>> >>> think is picking its default value i.e 1440 minutes so what do you >>> think >>> >>> what should I set if I am keeping my data for 30 days? >>> >>> >>> >>> Regards, >>> >>> Abhimanyu >>> >>> >>> >> >>> >> >>> > >>> >> >>
