Hello Dayo, This is a known issue, since today Kafka's log rolling / cleaning policy depends on the creation timestamp of the segment files, which could be modified upon partition migration / broker restart, it can cause the server to not honor the specified log cleaning config. Some more details ca be found at
KAFKA-881 KAFKA-979 Guozhang On Thu, Oct 2, 2014 at 10:48 AM, Dayo Oliyide <dayo.oliy...@gmail.com> wrote: > Hi, > > I've noticed an interesting behaviour which I hope someone can fully > explain. > I have 3 Kafka Node cluster with a setting of log.retention.hours=168 (7 > days) and log.segment.bytes=536870912. > > I recently restarted one of the nodes and it's uptime is now 3 days behind > than the other 2. > > After about 7 days I noticed that the other 2 nodes cleared out an equal > amount of stale logs/data, > but the restarted node didn't cleared out the same amount. The restarted > node only cleared out a > similar amount 3 days later. Generally now the restarted node seems to be 3 > days (free space wise) behind > the other 2. > > I noticed that certain partition log and index files on the restarted node > are different to the other 2. > Below is an example (Node C is the restarted one) > > > Node A - Topic Z Partition 12 > [ 488 Sep 26 11:47] 00000000000000046460.index > [ 1781829 Sep 24 12:09] 00000000000000046460.log > [ 10485760 Sep 28 22:23] 00000000000000046522.index > [ 1536693 Sep 28 22:23] 00000000000000046522.log > > Node B - Topic Z Partition 12 > [ 488 Sep 26 11:47] 00000000000000046460.index > [ 1781829 Sep 24 12:09] 00000000000000046460.log > [ 10485760 Sep 28 22:23] 00000000000000046522.index > [ 1536693 Sep 28 22:23] 00000000000000046522.log > > Node C - Topic Z Partition 12 > [ 10485760 Sep 28 22:23] 00000000000000046485.index > [ 2277311 Sep 28 22:23] 00000000000000046485.log > > I can see that Node C's base offset (log prefix) is in between the offsets > of the other logs on Node A & B, and that > suggests to me some partition 12 messages are on Node A & B but not on Node > C? > > I was hoping someone could help me figure out what's happening. > > Thanks > Dayo > -- -- Guozhang
