Hi Claudia
Just a sidenote, there is a combined policy for "compact, delete" which
deletes messages older than retention.ms and compacts newer ones if I
remember correctly.
It's still not really in the docs as it seems
https://kafka.apache.org/documentation/#topicconfigs
best regards
Patrik

On Mon, 1 Apr 2019 at 10:02, Claudia Wegmann <c.wegm...@kasasi.de> wrote:

> Hi,
>
> thanks for your reply.
>
> Also the groups were deleted month ago, there are still valid values
> there. So I guess deleting the group did not produce the tombstone record
> correctly. Your explanation made it clearer for me. I know I should keep
> cleanup policy compact in general. I decided to switch to policy for just
> long enough, so that these old records get deleted. I guess I could have
> produced the tombstone recods to be on the safe side.
>
> Thanks,
> Claudia
>
> -----Ursprüngliche Nachricht-----
> Von: Vincent Maurin <vincent.maurin...@gmail.com>
> Gesendet: Freitag, 29. März 2019 15:24
> An: users@kafka.apache.org
> Betreff: Re: Offsets of deleted consumer groups do not get deleted
> correctly
>
> Hi,
>
> You should keep the policy compact for the topic __consumer_offsets This
> topic stores for each group/topic/partition the offset consumed. As only
> the latest message for a group/topic/partition is relevant, the policy
> compact will keep only this message. When you delete a group, actually it
> will produce a tombstone to this topic (i.e body NULL). Then when the log
> compaction is running, it will definitively remove the tombstone.
> But to have an effective delete of the tombstones, keep in mind :
> * compaction runs only on rolled out segments
> * deletion of tombstone only occurs if the delete.retention.ms delay is
> expired
>
> Best regards
>
> On Fri, Mar 29, 2019 at 2:16 PM Claudia Wegmann <c.wegm...@kasasi.de>
> wrote:
>
> > Hey there,
> >
> > I've got the problem that the "__consumer_offsets" topic grows pretty
> > big over time. After some digging, I found offsets for consumer groups
> > that were deleted a long time ago still being present in the topic.
> > Many of them are offsets for console consumers, that have been deleted
> > with "kafka-consumer-groups.sh --delete --group ...".
> >
> > As far as I understand log cleaning, those offsets should have been
> > deleted a long time ago, because these consumers are no longer active.
> > When I query "kafka-consumer-groups.sh --bootstrap-server ...  --list"
> > I don't see those consumers either.
> >
> > Is there a bug in "kafka-consumer-groups.sh --delete --group ..." that
> > let's kafka hang on to those consumer groups?
> >
> > How can I get the log cleaner to delete these old offsets? Is there
> > another way than setting "cleanup.policy" to "delete"?
> >
> > Thanks for our help!
> >
> > Best,
> > Claudia
> >
>

Reply via email to