Added to the documentation. Topic level retentions is called retention.ms and can be found in http://kafka.apache.org/documentation.html#configuration
Thanks, Jun On Fri, Aug 15, 2014 at 12:32 PM, François Langelier <f.langel...@gmail.com> wrote: > UP! Can someone add the log.retention.hours in the documentation? > > And is the retention.hours or retention.minutes available for the by-topic > configuration? > > > > François Langelier > Étudiant en génie Logiciel - École de Technologie Supérieure > <http://www.etsmtl.ca/> > Capitaine Club Capra <http://capra.etsmtl.ca/> > VP-Communication - CS Games <http://csgames.org> 2014 > Jeux de Génie <http://www.jdgets.com/> 2011 à 2014 > Magistrat Fraternité du Piranha <http://fraternitedupiranha.com/> > Comité Organisateur Olympiades ÉTS 2012 > Compétition Québécoise d'Ingénierie 2012 - Compétition Senior > > > On Thu, Jul 10, 2014 at 11:16 PM, Virendra Pratap Singh < > vpsi...@yahoo-inc.com.invalid> wrote: > > > No worries, its working. My deployment was flawed and it was running > > 0.8.0, while I was under the impression that its running 0.8.1.1. > > After going to proper 0.8.1.1, the parameter started working properly. > > > > On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vpsi...@yahoo-inc.com> > > wrote: > > > > >Has someone really used : log.retention.minutes in kafka 0.8.1.1. > > >I have my full cluster running on 0.8.1.1 and the logs data is just not > > >getting cleaned up. > > > > > >And I see this message in kafka server.log > > >... > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not > valid > > >(kafka.utils.VerifiableProperties) > > >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not > valid > > >(kafka.utils.VerifiableProperties) > > >.... > > > > > >Virendra > > > > > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh" > > ><vpsi...@yahoo-inc.com.INVALID> wrote: > > > > > >>Thank you for pointing out to JIRA. > > >> > > >>On 7/9/14, 6:38 PM, "Jun Rao" <jun...@gmail.com> wrote: > > >> > > >>>This is being worked on in > > >>>https://issues.apache.org/jira/browse/KAFKA-1325 > > >>> > > >>>Thanks, > > >>> > > >>>Jun > > >>> > > >>> > > >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh < > > >>>vpsi...@yahoo-inc.com.invalid> wrote: > > >>> > > >>>> Well currently the log rollup is controlled via > > >>>> > > >>>> log.roll.hours > > >>>> Or > > >>>> log.segment.bytes > > >>>> > > >>>> Given that we now have support to log retention in minutes, I guess > it > > >>>> would be apt to have rollup also have capability to be available in > > >>>> minutes. Whom/where should I ask to have that coded in. > > >>>> > > >>>> One a similar note, I would genuinely want all the size and time > based > > >>>> parameters to be defined at bytes and ms level. This would make it > > >>>>generic > > >>>> enough to have user choose and define kind of setting they want and > > >>>>not > > >>>> require someone to go and change the code to support a new use case. > > >>>> Having hours/minutes in config will just cause > proliferation/bloating > > >>>>of > > >>>> config parameters. What is a good place to air this suggestion. > > >>>>Anyone? > > >>>> > > >>>> Virendra > > >>>> > > >>>> On 7/9/14, 7:21 AM, "Jun Rao" <jun...@gmail.com> wrote: > > >>>> > > >>>> >Actually, Kafka only removes old segments. The last (active) > segment > > >>>>is > > >>>> >never removed. So, f you want to have a 10 min retention, you need > to > > >>>> >configure log rolling such that log segments are rolled at least > > >>>>every > > >>>>10 > > >>>> >mins. > > >>>> > > > >>>> >Thanks, > > >>>> > > > >>>> >Jun > > >>>> > > > >>>> > > > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh < > > >>>> >vpsi...@yahoo-inc.com.invalid> wrote: > > >>>> > > > >>>> >> That's correct. The server where in I was running 0.8.1.1 was not > > >>>> >>honoring > > >>>> >> this parameter, despite the fact it was set in it > > >>>>server.properties. > > >>>> >> Not sure if this fact would play any role, the server which was > > >>>>running > > >>>> >> 0.8.0 was the leader for all the topics and partition in my > setup. > > >>>>And > > >>>> >>the > > >>>> >> second server running 0.8.1.1 has all the replicas (follower). > > >>>> >> > > >>>> >> Virendra > > >>>> >> > > >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: > > >>>> >> > > >>>> >> >Server properties should affect on only the local instance > > >>>>separately. > > >>>> >>Are > > >>>> >> >you saying the property is not honored even on the 0.8.1 > machines? > > >>>> >> > > > >>>> >> >Guozhang > > >>>> >> > > > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh < > > >>>> >> >vpsi...@yahoo-inc.com.invalid> wrote: > > >>>> >> > > > >>>> >> >> By setting this property > > >>>> >> >> log.retention.mins=10 > > >>>> >> >> in the server.properties file, which is passed as argument > when > > >>>> >>starting > > >>>> >> >> the broker. > > >>>> >> >> > > >>>> >> >> Virendra > > >>>> >> >> > > >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wangg...@gmail.com> > wrote: > > >>>> >> >> > > >>>> >> >> >How do you set the retention.minutes property? Is it through > > >>>> >>zk-based > > >>>> >> >> >topics tool? > > >>>> >> >> > > > >>>> >> >> >Guozhang > > >>>> >> >> > > > >>>> >> >> > > > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh < > > >>>> >> >> >vpsi...@yahoo-inc.com.invalid> wrote: > > >>>> >> >> > > > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 > broker > > >>>> >>0.8.0 > > >>>> >> >>and > > >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for > > >>>>partitions > > >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce > > >>>>the > > >>>> >>server > > >>>> >> >> >> properties settings as defined for that server. > > >>>> >> >> >> > > >>>> >> >> >> So this brings an interesting question, in case of > > >>>>heterogeneous > > >>>> >> >> >> environment (as is in my case, which system parameters will > > >>>>take > > >>>> >> >> >> preference/precedence). > > >>>> >> >> >> > > >>>> >> >> >> Virendra > > >>>> >> >> >> > > >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wangg...@gmail.com> > > >>>>wrote: > > >>>> >> >> >> > > >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1: > > >>>> >> >> >> > > > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918 > > >>>> >> >> >> > > > >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized. > > >>>> >> >> >> > > > >>>> >> >> >> >Guozhang > > >>>> >> >> >> > > > >>>> >> >> >> > > > >>>> >> >> >> > > > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh < > > >>>> >> >> >> >vpsi...@yahoo-inc.com.invalid> wrote: > > >>>> >> >> >> > > > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the > > >>>> >>broker1 > > >>>> >> >>is > > >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache > > >>>>release > > >>>> >> >>link. > > >>>> >> >> >> >> Directly using the tar ball, no local build used). > > >>>> >> >> >> >> > > >>>> >> >> >> >> I have set the log.retention.minutes=10. However the > > >>>>broker > > >>>>is > > >>>> >>not > > >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir > > >>>>at > > >>>> >>all. > > >>>> >> >> >> >> > > >>>> >> >> >> >> However when I set the log.retention.hours=1, then it > > >>>>starts > > >>>> >> >>cleaning > > >>>> >> >> >> >>the > > >>>> >> >> >> >> log. > > >>>> >> >> >> >> > > >>>> >> >> >> >> When I have the log.retention.minutes set in the > > >>>> >>server.properties > > >>>> >> >> >>then > > >>>> >> >> >> >>I > > >>>> >> >> >> >> see this logged in server.log: > > >>>> >> >> >> >> > > >>>> >> >> >> >> Š.. > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property > > >>>>log.retention.minutes > > >>>> >>is > > >>>> >> >>not > > >>>> >> >> >> >>valid > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties) > > >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property > > >>>>log.retention.minutes > > >>>> >>is > > >>>> >> >>not > > >>>> >> >> >> >>valid > > >>>> >> >> >> >> (kafka.utils.VerifiableProperties) > > >>>> >> >> >> >> ŠŠ > > >>>> >> >> >> >> > > >>>> >> >> >> >> > > >>>> >> >> >> >> I have set these properties too: > > >>>> >> >> >> >> > > >>>> >> >> >> >> log.cleaner.enable=true > > >>>> >> >> >> >> log.cleanup.policy=delete > > >>>> >> >> >> >> > > >>>> >> >> >> >> > > >>>> >> >> >> >> But I see similar warning logged for these properties > too. > > >>>> >> >> >> >> > > >>>> >> >> >> >> Regards, > > >>>> >> >> >> >> Virendra > > >>>> >> >> >> >> > > >>>> >> >> >> >> > > >>>> >> >> >> > > > >>>> >> >> >> > > > >>>> >> >> >> >-- > > >>>> >> >> >> >-- Guozhang > > >>>> >> >> >> > > >>>> >> >> >> > > >>>> >> >> > > > >>>> >> >> > > > >>>> >> >> >-- > > >>>> >> >> >-- Guozhang > > >>>> >> >> > > >>>> >> >> > > >>>> >> > > > >>>> >> > > > >>>> >> >-- > > >>>> >> >-- Guozhang > > >>>> >> > > >>>> >> > > >>>> > > >>>> > > >> > > > > > > > >
