@Abhimanyu You can try setting offset.retention = 30 (log.retention). At most, you will have a storage overhead of 5 million msgs per day * 30 (days) * 8 bytes (for each offset) = 1.2GB (not that much since you have a TB of hard disk)
On Sun, May 21, 2017 at 3:05 AM, kant kodali <kanth...@gmail.com> wrote: > Looking at that ticket and reading the comments it looks like one of the > concern is as follows. > > "offsets.retention.minutes is designed to handle the case that a consumer > group goes away forever. In that case, we don't want to store the offsets > for that group forever." > > This can simply be addressed by setting offset.retention == log.retention > by default right? In which case offset wont be stored forever even when > consumer group goes away forever. When the consumer group goes away forever > the upper bound to clean up offsets would be equal to log.retention. > > > > On Sun, May 21, 2017 at 2:19 AM, kant kodali <kanth...@gmail.com> wrote: > >> What is your average message size and network speed? >> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < >> abhimanyunagr...@gmail.com> wrote: >> >>> Hi Girish, >>> >>> I did not set any value for offsets.retention.minutes so therefore what I >>> think is picking its default value i.e 1440 minutes so what do you think >>> what should I set if I am keeping my data for 30 days? >>> >>> Regards, >>> Abhimanyu >>> >> >> >
