There is an issue about log files cannot be deleted in general https://issues.apache.org/jira/browse/KAFKA-1194 Someone suggested a solution for the deletion of regular topics (not compacted topics such as consumer offsets) requiring to make consumer offsets also a regular topic in addition to the fix for regular topics I created a fix also for the compaction part but didn't continue to make a pull request and fully verify the effect Currently Windows cannot be avoided in the scenario required by me
On Mon, Dec 31, 2018 at 8:36 AM Gwen Shapira <g...@confluent.io> wrote: > Not really... If you don't clean-up, you have to delete or the logs will > grow indefinitely. > > Is there a Jira for the windows issue? > Also, is there a way to avoid windows until this is resolved? Docker > containers perhaps? > > On Sun, Dec 30, 2018, 11:49 PM lk gen <lkge...@gmail.com wrote: > > > The original issue is that in windows the compaction cleanup is causing > the > > Kafka process to crash due to file handling, in order to avoid it, I > tried > > to disable the compaction cleanup, but that causes the consumer offsets > log > > to keep increasing, is there a way to work with zookeeper for consumer > > offsets with the latest versions of Kafka and consumers, or some other > way > > to bypass the cleanup of compacted logs ? > > > > On Sun, Dec 30, 2018 at 8:18 PM Gwen Shapira <g...@confluent.io> wrote: > > > > > Depending on how many consumer groups and partitions you have and how > > often > > > you commit, you risk either running out of disk space and/or deleting > > > commit information that you will need. > > > Either way, you will be storing lots of records you don't need. > > > > > > Only do this if there is no other solution to whatever the real issue > > > is... > > > > > > On Sun, Dec 30, 2018, 5:50 PM lk gen <lkge...@gmail.com wrote: > > > > > > > Hi, > > > > > > > > the consumer offsets Kafka internal topic is always created with a > > > > compact cleanup policy > > > > > > > > If altering the consumer offsets topic policy from compact to > delete > > > in a > > > > specific installed environment, will it cause problems ? will the > > > consumer > > > > still work if the consumer offsets are set to delete ? > > > > > > > > Thanks > > > > > > > > > >
