Re: Kafka log retention questions

2015-11-13 Thread Sandhu, Dilpreet
Thank you so much Gwen.This is really useful information :) On 11/13/15, 1:24 PM, "Gwen Shapira" wrote: >Mostly correct, yes. > >Just one additional point: >You configured the topic to retain data for 1s. However, there is also a >property for how often the broker will look for something to dele

Re: Kafka log retention questions

2015-11-13 Thread Gwen Shapira
Mostly correct, yes. Just one additional point: You configured the topic to retain data for 1s. However, there is also a property for how often the broker will look for something to delete: log.retention.check.interval.ms. This defaults to 5 minute. So unless you tweak it, the old messages will on

Re: Kafka log retention questions

2015-11-13 Thread Sandhu, Dilpreet
Thanks a lot Gwen and Raju. This information was really helpful. I used the following example:- kafka-topics.sh --zookeeper localhost:13003 --alter --topic MyTopic --config retention.ms=1000 And it works as expected. By setting its limit to a particular value, it prunes messages after that limit

Re: Kafka log retention questions

2015-11-13 Thread Raju Bairishetti
Adding some more info inline. On Fri, Nov 13, 2015 at 10:43 AM, Gwen Shapira wrote: > See answers inline > > On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet > wrote: > > > Hi all, > >I am new to Kafka usage. Here are some questions that I have in > > mind. Kindly help me understand it

Re: Kafka log retention questions

2015-11-12 Thread Gwen Shapira
See answers inline On Thu, Nov 12, 2015 at 2:53 PM, Sandhu, Dilpreet wrote: > Hi all, >I am new to Kafka usage. Here are some questions that I have in > mind. Kindly help me understand it better. If some questions make no sense > feel free to call it out. > 1. Is it possible to prune lo