Hi, Our kafka __consumer_offsets cleanup.policy set to delete, and retention.ms set to 86400000(one day), because we found the default compact mode using very large disk, and we change to delete mode many years ago. Now we become understanding kafka failover, and found some partitions under __conusmer_offsets size almost 28G(__consumer-offsets-12), and when a broker shutdown manually or crash, another broker owned election and become new partition leader, coordinator will read the 28G size offset and group metadata, it will take about 4-5 mins, and between this time, our consumers belong this coordinator can not commit the offsets, so can not consumer messages.
So we want to change the cleanup.policy from delete to compact, and I test this step on test environment, find the disk read speed almost 100M/s, and I'm afraid this will take large disk press to our production kafka cluster, And I want to consult can I delete some segment under __consumer_offsets-12 so , the compact segment will completed soon. We not want to delete segment to anther __consumer_offsets partitions only want to __consumer_offsets-12, so we don't want to using retention.ms to small time. Thanks.
