Jose,
How do you differentiate the compaction topics from the time retention
topics? Deleting a message due to compaction policy is a valid case
and users won't be interested in monitoring/reading those deleted messages.
Thanks,
Kamal
On Tue, Jul 23, 2019 at 4:00 AM Jose M <yoz...@gmail.com> wrote:
> Hi Colin,
>
> Thanks a lot for your feedback. Please note that I only propose to log when
> a message is lost this for a set of consumer groups, not as default
> behaviour for all consumer groups.
> But in fact, I agree with you that to log a line per message expired can be
> quite lot, and that is not the better way do it. I can propose to add a
> dedicated JMX metric of type counter "expired messages" per consumer group.
> What do you think ?
>
> About monitoring the lag to ensure that messages are not lost, I know that
> is what clients do, to set up alerting when the lag is above a threshold.
> But even if the alert is triggered, we dont know if messages have been lost
> or not. Implementing this KIP clients would know if something has been
> missed or not.
>
>
> Thanks,
>
>
> Jose
>
> On Mon, Jul 22, 2019 at 5:51 PM Colin McCabe <cmcc...@apache.org> wrote:
>
> > Hi Jose,
> >
> > One issue that I see here is that the number of log messages could be
> > huge. I've seen people create tens of thousands of consumer groups.
> > People can also have settings that create pretty small log files. A
> > message per log file per group could be quite a lot of messages.
> >
> > A log message on the broker is also not that useful for detecting bad
> > client behavior. People generally only look at the server logs after
> they
> > become aware that something is wrong through some other means.
> >
> > Perhaps the clients should just monitor their lag? There is a JMX metric
> > for this, which means it can be hooked into traditional metrics /
> reporting
> > systems.
> >
> > best,
> > Colin
> >
> >
> > On Mon, Jul 22, 2019, at 03:12, Jose M wrote:
> > > Hello,
> > >
> > > I didn't get any feedback on this small KIP-490
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+been+deleted
> > >.
> > > In summary, I propose a way to be noticed when messages are being
> > > removed
> > > due to retention policy, without being consumed by a given consumer
> > > group.
> > > It will be useful to realize that some important messages have been
> > > lost.
> > >
> > > As Im new to the codebase, I have technical questions about how to
> > achieve
> > > this, but before going deeper, I would like your feedback on the
> feature.
> > >
> > > Thanks a lot,
> > >
> > >
> > > Jose Morales
> > >
> > > On Sun, Jul 14, 2019 at 12:51 AM Jose M <yoz...@gmail.com> wrote:
> > >
> > > > Hello,
> > > >
> > > > I would like to know what do you think on KIP-490:
> > > >
> > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+been+deleted
> > > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-490%3A+log+when+consumer+groups+lose+a+message+because+offset+has+expired
> > >
> > > >
> > > >
> > > > Thanks a lot !
> > > > --
> > > > Jose M
> > > >
> > >
> > >
> > > --
> > > J
> > >
> >
>
>
> --
> J
>
