@Kant
Actually, after setting that configuration I deleted a topic having 100
partitions. After that, I checked through ./kafka-topic.sh --list command.
It was not showing that topic. Again I started producers and ran the
following command
*./kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe
--group <groupName- for topic marked for deletion>*
and got the following output:
*Note: This will only show information about consumers that use the Java
consumer API (non-ZooKeeper-based consumers).*
*[2017-05-24 07:17:48,444] WARN Error while fetching metadata with
correlation id 1 : {topicName=LEADER_NOT_AVAILABLE}
(org.apache.kafka.clients.NetworkClient)*
The same describe groups command is working fine for other groups.
So I am not able to get what can be the reason for such error.and how to
fix it.
Regards,
Abhimanyu
On Wed, May 24, 2017 at 12:27 PM, kant kodali <kanth...@gmail.com> wrote:
> @Abhimanyu
>
> 1) My guess is that topic offsets will remain for 30 days since that is the
> configuration you are explicitly setting and Kafka should respect that
> although I don't know for sure.
>
> 2) same as #1 offsets should remain to whatever time you specify.
>
> What is the problem with setting offsets.retention.minutes ==
> log.retention.hours
> ? Did it fix the problem you were facing before?
>
>
> On Tue, May 23, 2017 at 11:40 PM, Abhimanyu Nagrath <
> abhimanyunagr...@gmail.com> wrote:
>
> > Hi Kant,
> >
> > After setting this configuration offsets.retention.minutes . I am in
> doubt
> > about the two things
> > 1. If I am deleting a topic will that topic offsets would also get
> deleted
> > or will they present for 30 days?
> > 2. What will happen if for some topics my log.retention.hours = 168 and
> > offsets.retention.minutes= 1440 * 30 ?
> >
> > Regards,
> > Abhimanyu
> >
> > On Mon, May 22, 2017 at 4:09 PM, Abhimanyu Nagrath <
> > abhimanyunagr...@gmail.com> wrote:
> >
> > > @Kant I was going through the offset related configurations before
> > setting
> > > offsets.retention.minutes so came accross this configuration and
> thought
> > to
> > > ask whether this should also be tuned or not.
> > >
> > >
> > > Regards,
> > > Abhimanyu
> > >
> > >
> > >
> > >
> > > On Mon, May 22, 2017 at 2:24 PM, kant kodali <kanth...@gmail.com>
> wrote:
> > >
> > >> @Abhimanyu Why do you think you need to set that? Did you try setting
> > >> offsets.retention.minutes
> > >> = 1440 * 30 and still seeing duplicates?
> > >>
> > >> On Mon, May 22, 2017 at 12:37 AM, Abhimanyu Nagrath <
> > >> abhimanyunagr...@gmail.com> wrote:
> > >>
> > >> > Hi Girish ,
> > >> >
> > >> > Do I need to tune this configuration offsets.retention.check.interv
> > >> al.ms
> > >> > also . Please let me know if I need to tune any other configuration.
> > >> >
> > >> >
> > >> > Regards,
> > >> > Abhimanyu
> > >> >
> > >> > On Sun, May 21, 2017 at 8:01 PM, Girish Aher <girisha...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Yup, exactly as Kant said.
> > >> > > Also make sure that the retention of the offsets topic is an upper
> > >> bound
> > >> > > across all topics. So in this case, don't create any other topics
> in
> > >> the
> > >> > > future with retention of more than 30 days or otherwise they may
> > have
> > >> the
> > >> > > same problem too.
> > >> > >
> > >> > > On May 21, 2017 03:25, "Abhimanyu Nagrath" <
> > >> abhimanyunagr...@gmail.com>
> > >> > > wrote:
> > >> > >
> > >> > >> Hi Kant,
> > >> > >>
> > >> > >> Thanks for the suggestion.
> > >> > >>
> > >> > >>
> > >> > >> Regards,
> > >> > >> Abhimanyu
> > >> > >>
> > >> > >> On Sun, May 21, 2017 at 3:44 PM, kant kodali <kanth...@gmail.com
> >
> > >> > wrote:
> > >> > >>
> > >> > >>> @Abhimanyu You can try setting offset.retention = 30
> > >> (log.retention).
> > >> > At
> > >> > >>> most, you will have a storage overhead of 5 million msgs per
> day *
> > >> 30
> > >> > >>> (days) * 8 bytes (for each offset) = 1.2GB (not that much since
> > you
> > >> > have
> > >> > >>> a
> > >> > >>> TB of hard disk)
> > >> > >>>
> > >> > >>> On Sun, May 21, 2017 at 3:05 AM, kant kodali <
> kanth...@gmail.com>
> > >> > wrote:
> > >> > >>>
> > >> > >>> > Looking at that ticket and reading the comments it looks like
> > one
> > >> of
> > >> > >>> the
> > >> > >>> > concern is as follows.
> > >> > >>> >
> > >> > >>> > "offsets.retention.minutes is designed to handle the case
> that a
> > >> > >>> consumer
> > >> > >>> > group goes away forever. In that case, we don't want to store
> > the
> > >> > >>> offsets
> > >> > >>> > for that group forever."
> > >> > >>> >
> > >> > >>> > This can simply be addressed by setting offset.retention ==
> > >> > >>> log.retention
> > >> > >>> > by default right? In which case offset wont be stored forever
> > even
> > >> > when
> > >> > >>> > consumer group goes away forever. When the consumer group goes
> > >> away
> > >> > >>> forever
> > >> > >>> > the upper bound to clean up offsets would be equal to
> > >> log.retention.
> > >> > >>> >
> > >> > >>> >
> > >> > >>> >
> > >> > >>> > On Sun, May 21, 2017 at 2:19 AM, kant kodali <
> > kanth...@gmail.com>
> > >> > >>> wrote:
> > >> > >>> >
> > >> > >>> >> What is your average message size and network speed?
> > >> > >>> >>
> > >> > >>> >> On Sun, May 21, 2017 at 2:04 AM, Abhimanyu Nagrath <
> > >> > >>> >> abhimanyunagr...@gmail.com> wrote:
> > >> > >>> >>
> > >> > >>> >>> Hi Girish,
> > >> > >>> >>>
> > >> > >>> >>> I did not set any value for offsets.retention.minutes so
> > >> therefore
> > >> > >>> what I
> > >> > >>> >>> think is picking its default value i.e 1440 minutes so what
> do
> > >> you
> > >> > >>> think
> > >> > >>> >>> what should I set if I am keeping my data for 30 days?
> > >> > >>> >>>
> > >> > >>> >>> Regards,
> > >> > >>> >>> Abhimanyu
> > >> > >>> >>>
> > >> > >>> >>
> > >> > >>> >>
> > >> > >>> >
> > >> > >>>
> > >> > >>
> > >> > >>
> > >> >
> > >>
> > >
> > >
> >
>
