Thanks Damian, I worked. I changed the StreamsConfig.WINDOW_STORE_ CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG and was able to reduce the retention time of the changelog topic.
-Sameer. On Mon, Oct 30, 2017 at 9:38 PM, Damian Guy <damian....@gmail.com> wrote: > The retention for the joins is as specified above. With until set to 240 > and StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG > set to 1 day (the default) this would be 100800000 > > For plain key value stores, there should be no retention period as the > topics are compacted only. > > On Mon, 30 Oct 2017 at 15:48 Sameer Kumar <sam.kum.w...@gmail.com> wrote: > > > Actually I am using Key Value store, I do use join as part of my > DAG(until > > for the same has been set at 240 mins). The sink processor is key-value, > is > > there any option to control it. > > > > -Sameer. > > > > On Mon, Oct 30, 2017 at 6:33 PM, Damian Guy <damian....@gmail.com> > wrote: > > > > > The topics in question are both changelogs for window stores. The > > retention > > > period for them is calculated as the Window retention period, which is > > the > > > value that is passed to `JoinWindows.until(...)` (default is 1 day) > plus > > > the value of the config > > > StreamsConfig.WINDOW_STORE_CHANGE_LOG_ADDITIONAL_RETENTION_MS_CONFIG ( > > > default is 1 day) > > > > > > > > > > > > On Mon, 30 Oct 2017 at 12:49 Sameer Kumar <sam.kum.w...@gmail.com> > > wrote: > > > > > > > Hi, > > > > > > > > I have configured my settings to be the following:- > > > > > > > > log.retention.hours=3 > > > > delete.topic.enable=true > > > > delete.retention.ms=10800000 > > > > min.cleanable.dirty.ratio=0.20 > > > > segment.ms=180000 > > > > > > > > Howsoever, the changelog topic created as part of stream has the > > > > rentention.ms to be 100800000, the source topic has it to be 3 > hours. > > > > > > > > [root@dmpkafka6591 kafka-11-single]# bin/kafka-topics.sh --describe > > > > --zookeeper 172.29.65.91:2181 --topic > > > > c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog > > > > Topic:c-7-aq7-KSTREAM-JOINOTHER-0000000019-store-changelog > > > > PartitionCount:60 ReplicationFactor:1 Configs:retention.ms > = > > > > 100800000,cleanup.policy=delete,compact > > > > > > > > Can someone please explain this behavior. > > > > > > > > -Sameer. > > > > > > > > > >
