Re: consumer offsets topic cleanup policy

2018-12-31 Thread lk gen
There is an issue about log files cannot be deleted in general https://issues.apache.org/jira/browse/KAFKA-1194 Someone suggested a solution for the deletion of regular topics (not compacted topics such as consumer offsets) requiring to make consumer offsets also a regular topic in addition to the

Re: consumer offsets topic cleanup policy

2018-12-30 Thread Gwen Shapira
Not really... If you don't clean-up, you have to delete or the logs will grow indefinitely. Is there a Jira for the windows issue? Also, is there a way to avoid windows until this is resolved? Docker containers perhaps? On Sun, Dec 30, 2018, 11:49 PM lk gen The original issue is that in windows

Re: consumer offsets topic cleanup policy

2018-12-30 Thread lk gen
The original issue is that in windows the compaction cleanup is causing the Kafka process to crash due to file handling, in order to avoid it, I tried to disable the compaction cleanup, but that causes the consumer offsets log to keep increasing, is there a way to work with zookeeper for consumer o

Re: consumer offsets topic cleanup policy

2018-12-30 Thread Gwen Shapira
Depending on how many consumer groups and partitions you have and how often you commit, you risk either running out of disk space and/or deleting commit information that you will need. Either way, you will be storing lots of records you don't need. Only do this if there is no other solution to wh