Hi Kafka people,
I've configured my brokers to use one hour (3600000 ms) for the
min.compaction.lag.ms.
In the logs I see that this is picked up in the 'created log' line. I add a
bit over a million messages, and I see the new log segments appearing
('Rolling new log segment') but after a few minutes I see that the
compaction kicks in, long before an hour has passed.
Is there anything weird in my config? Is something missing?
I'm using 0.10.2.1
regards, Frank
[2017-05-29 13:09:32,498] INFO Created log for partition
[testsetup-matchteammemberattrgroup,0] in /data with properties
{compression.type -> producer, message.format.version -> 0.10.2-IV0,
file.delete.delay.ms -> 60000, max.message.bytes -> 8000000,
min.compaction.lag.ms -> 3600000, message.timestamp.type -> CreateTime,
min.insync.replicas -> 1, segment.jitter.ms -> 0, preallocate -> false,
min.cleanable.dirty.ratio -> 0.5, index.interval.bytes -> 4096,
unclean.leader.election.enable -> true, retention.bytes -> -1,
delete.retention.ms -> 86400000, cleanup.policy -> compact, flush.ms ->
9223372036854775807, segment.ms -> 604800000, segment.bytes -> 10485760,
retention.ms -> 6048000000, message.timestamp.difference.max.ms ->
9223372036854775807, segment.index.bytes -> 10485760, flush.messages ->
9223372036854775807}. (kafka.log.LogManager)
[2017-05-29 13:09:32,501] INFO Partition
[testsetup-matchteammemberattrgroup,0] on broker 1: No checkpointed
highwatermark is found for partition testsetup-matchteammemberattrgroup-0
(kafka.cluster.Partition)
[2017-05-29 13:14:22,180] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 1 ms. (kafka.log.Log)
[2017-05-29 13:14:27,286] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 0 ms. (kafka.log.Log)
[2017-05-29 13:14:31,308] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 0 ms. (kafka.log.Log)
[2017-05-29 13:14:35,206] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 0 ms. (kafka.log.Log)
[2017-05-29 13:14:37,374] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 1 ms. (kafka.log.Log)
[2017-05-29 13:14:41,666] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 0 ms. (kafka.log.Log)
[2017-05-29 13:14:44,211] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 0 ms. (kafka.log.Log)
[2017-05-29 13:14:49,112] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 1 ms. (kafka.log.Log)
[2017-05-29 13:14:51,851] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 1 ms. (kafka.log.Log)
[2017-05-29 13:14:54,497] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 1 ms. (kafka.log.Log)
[2017-05-29 13:15:00,314] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 1 ms. (kafka.log.Log)
[2017-05-29 13:15:03,453] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 0 ms. (kafka.log.Log)
[2017-05-29 13:15:07,795] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 1 ms. (kafka.log.Log)
[2017-05-29 13:15:11,071] INFO Rolled new log segment for
'testsetup-matchteammemberattrgroup-0' in 2 ms. (kafka.log.Log)
[2017-05-29 13:15:31,110] INFO Deleting segment 0 from log
testsetup-matchteammemberattrgroup-0. (kafka.log.Log)
[2017-05-29 13:15:31,118] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000000000.index.deleted
(kafka.log.OffsetIndex)
[2017-05-29 13:15:31,118] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000000000.timeindex.deleted
(kafka.log.TimeIndex)
[2017-05-29 13:15:31,291] INFO Deleting segment 22897 from log
testsetup-matchteammemberattrgroup-0. (kafka.log.Log)
[2017-05-29 13:15:31,299] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000022897.index.deleted
(kafka.log.OffsetIndex)
[2017-05-29 13:15:31,299] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000022897.timeindex.deleted
(kafka.log.TimeIndex)
[2017-05-29 13:15:32,103] INFO Deleting segment 0 from log
testsetup-matchteammemberattrgroup-0. (kafka.log.Log)
[2017-05-29 13:15:32,104] INFO Deleting segment 22897 from log
testsetup-matchteammemberattrgroup-0. (kafka.log.Log)
[2017-05-29 13:15:32,105] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000000000.index.deleted
(kafka.log.OffsetIndex)
[2017-05-29 13:15:32,105] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000000000.timeindex.deleted
(kafka.log.TimeIndex)
[2017-05-29 13:15:32,107] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000022897.index.deleted
(kafka.log.OffsetIndex)
[2017-05-29 13:15:32,108] INFO Deleting index
/data/testsetup-matchteammemberattrgroup-0/00000000000000022897.timeindex.deleted
(kafka.log.TimeIndex)
[2017-05-29 13:15:32,175] INFO Deleting segment 45782 from log
testsetup-matchteammemberattrgroup-0. (kafka
