I tried setting segment.ms=3600000 and retention.ms=60000 and the segments were still not deleting after several hours. Also, this is a high volume topic, so log.segment.bytes=1073741824 is easily met within an hour.
What seems to have worked was to shutdown all Kafka brokers, manually deleting all data for that topic across all brokers, and restarting the brokers. Since then they have been deleting data according to the retention.ms setting. I am now monitoring the logs and will slowly increase retention.ms back to the desired 2 days. We have ran Kafka in this config for three years and this is the first time this has happened. No idea what triggered the issue and no errors/warnings were in the Kafka logs. On Wed, May 30, 2018 at 12:50 AM, Shantanu Deshmukh <shantanu...@gmail.com> wrote: > Hey, > > You should try setting topic level config by doing kafka-topics.sh --alter > --topic <topic-name> --config <config>=<value> --zookeeper <ZK host:port> > > Make sure you also set segment.ms for topics which are not that populous. > This setting specifies amount of time after which a new segment is rolled. > So Kafka deletes only those messages which lie in a segment which is old or > full. Basically Kafka doesn't touch current segment. So if we roll soon > enough changes of messages in it getting eligible for retention.ms setting > increases. I am not fully sure what effect it might have on cluster > resources if segment.ms value is kept too low, as broker might spend too > much resources just rolling many segments. So keep it some reasonable value. > > On Tue, May 29, 2018 at 9:31 PM Thomas Hays <hay...@gmail.com> wrote: > >> A single topic does not appear to be honoring the retention.ms >> setting. Three other topics (plus __consumer_offsets) on the Kafka >> instance are deleting segments normally. >> >> Kafka version: 2.12-0.10.2.1 >> OS: CentOS 7 >> Java: openjdk version "1.8.0_161" >> Zookeeper: 3.4.6 >> >> Retention settings (from kafka-topics.sh describe): Topic:elk >> PartitionCount:50 ReplicationFactor:2 Configs:retention.ms=7200000 >> >> Other config settings from server.properties >> >> log.retention.hours=48 >> log.segment.bytes=1073741824 >> log.retention.check.interval.ms=300000 >> >> Looking in the data directory, I see multiple segment files older than 48 >> hours: >> >> -rw-r--r-- 1 root root 1073676782 May 26 20:16 00000000004713142447.log >> -rw-r--r-- 1 root root 1073105605 May 26 20:18 00000000004715239774.log >> -rw-r--r-- 1 root root 1072907965 May 26 20:20 00000000004717450325.log >> >> Current date/time on server: Tue May 29 10:51:49 CDT 2018 >> >> This issue appears on all Kafka brokers and I have tried multiple >> rolling restarts of all Kafka brokers and the issue remains. These >> servers stopped deleting segments for this topic on May 15. This does >> not correlate to any known config change. I have found no >> error/warning messages in the logs to indicate a problem. >> >> What am I missing? Thank you. >>
