Hi Claudia Just a sidenote, there is a combined policy for "compact, delete" which deletes messages older than retention.ms and compacts newer ones if I remember correctly. It's still not really in the docs as it seems https://kafka.apache.org/documentation/#topicconfigs best regards Patrik
On Mon, 1 Apr 2019 at 10:02, Claudia Wegmann <c.wegm...@kasasi.de> wrote: > Hi, > > thanks for your reply. > > Also the groups were deleted month ago, there are still valid values > there. So I guess deleting the group did not produce the tombstone record > correctly. Your explanation made it clearer for me. I know I should keep > cleanup policy compact in general. I decided to switch to policy for just > long enough, so that these old records get deleted. I guess I could have > produced the tombstone recods to be on the safe side. > > Thanks, > Claudia > > -----Ursprüngliche Nachricht----- > Von: Vincent Maurin <vincent.maurin...@gmail.com> > Gesendet: Freitag, 29. März 2019 15:24 > An: users@kafka.apache.org > Betreff: Re: Offsets of deleted consumer groups do not get deleted > correctly > > Hi, > > You should keep the policy compact for the topic __consumer_offsets This > topic stores for each group/topic/partition the offset consumed. As only > the latest message for a group/topic/partition is relevant, the policy > compact will keep only this message. When you delete a group, actually it > will produce a tombstone to this topic (i.e body NULL). Then when the log > compaction is running, it will definitively remove the tombstone. > But to have an effective delete of the tombstones, keep in mind : > * compaction runs only on rolled out segments > * deletion of tombstone only occurs if the delete.retention.ms delay is > expired > > Best regards > > On Fri, Mar 29, 2019 at 2:16 PM Claudia Wegmann <c.wegm...@kasasi.de> > wrote: > > > Hey there, > > > > I've got the problem that the "__consumer_offsets" topic grows pretty > > big over time. After some digging, I found offsets for consumer groups > > that were deleted a long time ago still being present in the topic. > > Many of them are offsets for console consumers, that have been deleted > > with "kafka-consumer-groups.sh --delete --group ...". > > > > As far as I understand log cleaning, those offsets should have been > > deleted a long time ago, because these consumers are no longer active. > > When I query "kafka-consumer-groups.sh --bootstrap-server ... --list" > > I don't see those consumers either. > > > > Is there a bug in "kafka-consumer-groups.sh --delete --group ..." that > > let's kafka hang on to those consumer groups? > > > > How can I get the log cleaner to delete these old offsets? Is there > > another way than setting "cleanup.policy" to "delete"? > > > > Thanks for our help! > > > > Best, > > Claudia > > >
