I tried running reassignment on the topic, but that didn't help. I had to restart the broker for it to release the file handlers, then manually delete.
On Fri, Oct 28, 2016 at 6:25 PM, James Brown <jbr...@easypost.com> wrote: > I was having this problem with one of my __consumer_offsets partitions; I > used reassignment to move the large partition onto a different set of > machines (which forced the cleaner to run through them again) and after the > new machines finished replicating, the partition was back down from 41GB to > a nice trim 38MB. > > On Fri, Oct 28, 2016 at 1:00 PM, Chi Hoang <chi.ho...@zuora.com> wrote: > > > Hi, > > We have a 3-node cluster that is running 0.9.0.1, and recently saw that > the > > "__consumer_offsets" topic on one of the nodes seems really skewed with > > disk usage that looks like: > > > > 73G ./__consumer_offsets-10 > > 0 ./__consumer_offsets-7 > > 0 ./__consumer_offsets-4 > > 0 ./__consumer_offsets-1 > > 0 ./__consumer_offsets-49 > > 19G ./__consumer_offsets-46 > > 0 ./__consumer_offsets-43 > > 0 ./__consumer_offsets-40 > > > > > > > > This goes on for all 50 partitions. Upon inspection, we saw that a lot > of > > the log files were old: > > > > ll __consumer_offsets-10 > > total 76245192 > > -rw-r--r-- 1 root root 0 Oct 7 20:14 00000000000000000000.index > > -rw-r--r-- 1 root root 901 Oct 7 20:14 00000000000000000000.log > > -rw-r--r-- 1 root root 157904 Oct 7 22:15 00000000000907046457.index > > -rw-r--r-- 1 root root 104855056 Oct 7 22:15 00000000000907046457.log > > -rw-r--r-- 1 root root 157904 Oct 7 22:51 00000000000909543421.index > > -rw-r--r-- 1 root root 104853568 Oct 7 22:51 00000000000909543421.log > > -rw-r--r-- 1 root root 157904 Oct 7 23:27 00000000000910806717.index > > -rw-r--r-- 1 root root 104853568 Oct 7 23:27 00000000000910806717.log > > > > > > We are using default parameters as it pertains to offset management, and > > our config output includes the following entries: > > > > log.cleaner.enable = true > > > > offsets.retention.minutes = 1440 > > > > > > I tried looking through the issues on JIRA but didn't see a reported > > issue. Does anyone know what's going on, and how I can fix this? > > > > Thanks. > > > > > > -- > James Brown > Engineer >
