@Kant Actually, after setting that configuration I deleted a topic having 100 partitions. After that, I checked through ./kafka-topic.sh --list command. It was not showing that topic. Again I started producers and ran the following command
*./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group <groupName- for topic marked for deletion>* and got the following output: *Note: This will only show information about consumers that use the Java consumer API (non-ZooKeeper-based consumers).* *[2017-05-24 07:17:48,444] WARN Error while fetching metadata with correlation id 1 : {topicName=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)* The same describe groups command is working fine for other groups. So I am not able to get what can be the reason for such error.and how to fix it. Regards, Abhimanyu On Wed, May 24, 2017 at 12:27 PM, kant kodali <kanth...@gmail.com> wrote: > @Abhimanyu > > 1) My guess is that topic offsets will remain for 30 days since that is the > configuration you are explicitly setting and Kafka should respect that > although I don't know for sure. > > 2) same as #1 offsets should remain to whatever time you specify. > > What is the problem with setting offsets.retention.minutes == > log.retention.hours > ? Did it fix the problem you were facing before? > > > On Tue, May 23, 2017 at 11:40 PM, Abhimanyu Nagrath < > abhimanyunagr...@gmail.com> wrote: > > > Hi Kant, > > > > After setting this configuration offsets.retention.minutes . I am in > doubt > > about the two things > > 1. If I am deleting a topic will that topic offsets would also get > deleted > > or will they present for 30 days? > > 2. What will happen if for some topics my log.retention.hours = 168 and > > offsets.retention.minutes= 1440 * 30 ? > > > > Regards, > > Abhimanyu > > > > On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath < > > abhimanyunagr...@gmail.com> wrote: > > > > > @Kant I was going through the offset related configurations before > > setting > > > offsets.retention.minutes so came accross this configuration and > thought > > to > > > ask whether this should also be tuned or not. > > > > > > > > > Regards, > > > Abhimanyu > > > > > > > > > > > > > > > On Mon, May 22, 2017 at 2:24 PM, kant kodali <kanth...@gmail.com> > wrote: > > > > > >> @Abhimanyu Why do you think you need to set that? Did you try setting > > >> offsets.retention.minutes > > >> = 1440 * 30 and still seeing duplicates? > > >> > > >> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath < > > >> abhimanyunagr...@gmail.com> wrote: > > >> > > >> > Hi Girish , > > >> > > > >> > Do I need to tune this configuration offsets.retention.check.interv > > >> al.ms > > >> > also . Please let me know if I need to tune any other configuration. > > >> > > > >> > > > >> > Regards, > > >> > Abhimanyu > > >> > > > >> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girisha...@gmail.com> > > >> wrote: > > >> > > > >> > > Yup, exactly as Kant said. > > >> > > Also make sure that the retention of the offsets topic is an upper > > >> bound > > >> > > across all topics. So in this case, don't create any other topics > in > > >> the > > >> > > future with retention of more than 30 days or otherwise they may > > have > > >> the > > >> > > same problem too. > > >> > > > > >> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" < > > >> abhimanyunagr...@gmail.com> > > >> > > wrote: > > >> > > > > >> > >> Hi Kant, > > >> > >> > > >> > >> Thanks for the suggestion. > > >> > >> > > >> > >> > > >> > >> Regards, > > >> > >> Abhimanyu > > >> > >> > > >> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com > > > > >> > wrote: > > >> > >> > > >> > >>> @Abhimanyu You can try setting offset.retention = 30 > > >> (log.retention). > > >> > At > > >> > >>> most, you will have a storage overhead of 5 million msgs per > day * > > >> 30 > > >> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since > > you > > >> > have > > >> > >>> a > > >> > >>> TB of hard disk) > > >> > >>> > > >> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali < > kanth...@gmail.com> > > >> > wrote: > > >> > >>> > > >> > >>> > Looking at that ticket and reading the comments it looks like > > one > > >> of > > >> > >>> the > > >> > >>> > concern is as follows. > > >> > >>> > > > >> > >>> > "offsets.retention.minutes is designed to handle the case > that a > > >> > >>> consumer > > >> > >>> > group goes away forever. In that case, we don't want to store > > the > > >> > >>> offsets > > >> > >>> > for that group forever." > > >> > >>> > > > >> > >>> > This can simply be addressed by setting offset.retention == > > >> > >>> log.retention > > >> > >>> > by default right? In which case offset wont be stored forever > > even > > >> > when > > >> > >>> > consumer group goes away forever. When the consumer group goes > > >> away > > >> > >>> forever > > >> > >>> > the upper bound to clean up offsets would be equal to > > >> log.retention. > > >> > >>> > > > >> > >>> > > > >> > >>> > > > >> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali < > > kanth...@gmail.com> > > >> > >>> wrote: > > >> > >>> > > > >> > >>> >> What is your average message size and network speed? > > >> > >>> >> > > >> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath < > > >> > >>> >> abhimanyunagr...@gmail.com> wrote: > > >> > >>> >> > > >> > >>> >>> Hi Girish, > > >> > >>> >>> > > >> > >>> >>> I did not set any value for offsets.retention.minutes so > > >> therefore > > >> > >>> what I > > >> > >>> >>> think is picking its default value i.e 1440 minutes so what > do > > >> you > > >> > >>> think > > >> > >>> >>> what should I set if I am keeping my data for 30 days? > > >> > >>> >>> > > >> > >>> >>> Regards, > > >> > >>> >>> Abhimanyu > > >> > >>> >>> > > >> > >>> >> > > >> > >>> >> > > >> > >>> > > > >> > >>> > > >> > >> > > >> > >> > > >> > > > >> > > > > > > > > >