Re: consumer offsets topic cleanup policy

2018-12-31 Thread lk gen
There is an issue about log files cannot be deleted in general https://issues.apache.org/jira/browse/KAFKA-1194 Someone suggested a solution for the deletion of regular topics (not compacted topics such as consumer offsets) requiring to make consumer offsets also a regular topic in addition to the

Re: consumer offsets topic cleanup policy

2018-12-30 Thread Gwen Shapira
Not really... If you don't clean-up, you have to delete or the logs will grow indefinitely. Is there a Jira for the windows issue? Also, is there a way to avoid windows until this is resolved? Docker containers perhaps? On Sun, Dec 30, 2018, 11:49 PM lk gen The original issue is that in windows

Re: consumer offsets topic cleanup policy

2018-12-30 Thread lk gen
The original issue is that in windows the compaction cleanup is causing the Kafka process to crash due to file handling, in order to avoid it, I tried to disable the compaction cleanup, but that causes the consumer offsets log to keep increasing, is there a way to work with zookeeper for consumer o

Re: consumer offsets topic cleanup policy

2018-12-30 Thread Gwen Shapira
Depending on how many consumer groups and partitions you have and how often you commit, you risk either running out of disk space and/or deleting commit information that you will need. Either way, you will be storing lots of records you don't need. Only do this if there is no other solution to wh

consumer offsets topic cleanup policy

2018-12-30 Thread lk gen
Hi, the consumer offsets Kafka internal topic is always created with a compact cleanup policy If altering the consumer offsets topic policy from compact to delete in a specific installed environment, will it cause problems ? will the consumer still work if the consumer offsets are set to dele