Thank you so much Gwen.This is really useful information :) On 11/13/15, 1:24 PM, "Gwen Shapira" <g...@confluent.io> wrote:
>Mostly correct, yes. > >Just one additional point: >You configured the topic to retain data for 1s. However, there is also a >property for how often the broker will look for something to delete: >log.retention.check.interval.ms. >This defaults to 5 minute. So unless you tweak it, the old messages will >only be deleted every 5 minutes. > >Gwen > > >On Fri, Nov 13, 2015 at 11:07 AM, Sandhu, Dilpreet <dsan...@illumina.com> >wrote: > >> Thanks a lot Gwen and Raju. This information was really helpful. >> >> I used the following example:- >> kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic >> --config retention.ms=1000 >> >> >> And it works as expected. >> By setting its limit to a particular value, it prunes messages after >>that >> limit. I just want to make sure I am looking at right property. >> >> >> Kindly provide me with feedback, if I am on right direction. >> Thanks once again. >> >> Your help is much appreciated. >> Best regards, >> Dilpreet >> >> >> >> On 11/13/15, 1:24 AM, "Raju Bairishetti" <raju....@gmail.com> wrote: >> >> >Adding some more info inline. >> > >> >On Fri, Nov 13, 2015 at 10:43 AM, Gwen Shapira <g...@confluent.io> >>wrote: >> > >> >> See answers inline >> >> >> >> On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet >><dsan...@illumina.com >> > >> >> wrote: >> >> >> >> > Hi all, >> >> > I am new to Kafka usage. Here are some questions that I >>have in >> >> > mind. Kindly help me understand it better. If some questions make >>no >> >> sense >> >> > feel free to call it out. >> >> > 1. Is it possible to prune log offsets (messages)older than >>certain >> >>date >> >> > in a partition? >> >> > >> >> >> >> You can configure the retention time for a topic, all older messages >> >>will >> >> be pruned. This configuration is at topic level, not for a specific >> >> partition. >> >> >> > >> >*IMO, we can set the retention in bytes(**retention.bytes**) at >>partition >> >level also. Whenever the partition log grows more than configured >> >retention >> >bytes then it deletes the older messages from the partition log. >>**Please >> >correct me if I am wrong.* >> > >> >> >> >> >> >> > 2. Will Kafka delete a topic regardless of unconsumed messages, if >>log >> >> > retention hours limit is met? >> >> > >> >> >> >> Yes >> >> >> > >> > Kafka does not care & does not know whether the message is >>consumed or >> >not. Kafka provides a mechanism to store all messages from multiple >> >producer machines to a central location. >> > >> >> >> >> >> >> > 3. Is there a way to just delete messages in Kafka? >> >> > >> >> >> >> Normally, we are only deleting messages when they are old or when >>size >> >> limit is exceeded. >> >> There is a special "compaction" mode where the last message for any >>key >> >>is >> >> stored forever. In this mode you can delete messages by sending a new >> >> message with that key and a "null" value. However, this also just >>takes >> >> place after a certain amount of time. >> >> >> >> >> >> > >> >> > Any help will be much appreciated. >> >> > >> >> > Thanks. >> >> > Best regards, >> >> > Dilpreet >> >> > >> >> >> > >> > >> > >> >-- >> >Thanks >> >Raju Bairishetti, >> > >> >www.inmobi.com >> > >> > >> > >> > >> >*2012 ME Best Mobile Ad Network Award Winner >> > >> >[image: http://www.inmobi.com/press-releases/files/2013/02/MIT.png] >> >>
