Mostly correct, yes. Just one additional point: You configured the topic to retain data for 1s. However, there is also a property for how often the broker will look for something to delete: log.retention.check.interval.ms. This defaults to 5 minute. So unless you tweak it, the old messages will only be deleted every 5 minutes.
Gwen On Fri, Nov 13, 2015 at 11:07 AM, Sandhu, Dilpreet <dsan...@illumina.com> wrote: > Thanks a lot Gwen and Raju. This information was really helpful. > > I used the following example:- > kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic > --config retention.ms=1000 > > > And it works as expected. > By setting its limit to a particular value, it prunes messages after that > limit. I just want to make sure I am looking at right property. > > > Kindly provide me with feedback, if I am on right direction. > Thanks once again. > > Your help is much appreciated. > Best regards, > Dilpreet > > > > On 11/13/15, 1:24 AM, "Raju Bairishetti" <raju....@gmail.com> wrote: > > >Adding some more info inline. > > > >On Fri, Nov 13, 2015 at 10:43 AM, Gwen Shapira <g...@confluent.io> wrote: > > > >> See answers inline > >> > >> On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet <dsan...@illumina.com > > > >> wrote: > >> > >> > Hi all, > >> > I am new to Kafka usage. Here are some questions that I have in > >> > mind. Kindly help me understand it better. If some questions make no > >> sense > >> > feel free to call it out. > >> > 1. Is it possible to prune log offsets (messages)older than certain > >>date > >> > in a partition? > >> > > >> > >> You can configure the retention time for a topic, all older messages > >>will > >> be pruned. This configuration is at topic level, not for a specific > >> partition. > >> > > > >*IMO, we can set the retention in bytes(**retention.bytes**) at partition > >level also. Whenever the partition log grows more than configured > >retention > >bytes then it deletes the older messages from the partition log. **Please > >correct me if I am wrong.* > > > >> > >> > >> > 2. Will Kafka delete a topic regardless of unconsumed messages, if log > >> > retention hours limit is met? > >> > > >> > >> Yes > >> > > > > Kafka does not care & does not know whether the message is consumed or > >not. Kafka provides a mechanism to store all messages from multiple > >producer machines to a central location. > > > >> > >> > >> > 3. Is there a way to just delete messages in Kafka? > >> > > >> > >> Normally, we are only deleting messages when they are old or when size > >> limit is exceeded. > >> There is a special "compaction" mode where the last message for any key > >>is > >> stored forever. In this mode you can delete messages by sending a new > >> message with that key and a "null" value. However, this also just takes > >> place after a certain amount of time. > >> > >> > >> > > >> > Any help will be much appreciated. > >> > > >> > Thanks. > >> > Best regards, > >> > Dilpreet > >> > > >> > > > > > > > >-- > >Thanks > >Raju Bairishetti, > > > >www.inmobi.com > > > > > > > > > >*2012 ME Best Mobile Ad Network Award Winner > > > >[image: http://www.inmobi.com/press-releases/files/2013/02/MIT.png] > >
