No worries, its working. My deployment was flawed and it was running 0.8.0, while I was under the impression that its running 0.8.1.1. After going to proper 0.8.1.1, the parameter started working properly.
On 7/10/14, 2:18 PM, "Virendra Pratap Singh" <vpsi...@yahoo-inc.com> wrote: >Has someone really used : log.retention.minutes in kafka 0.8.1.1. >I have my full cluster running on 0.8.1.1 and the logs data is just not >getting cleaned up. > >And I see this message in kafka server.log >... >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid >(kafka.utils.VerifiableProperties) >[2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid >(kafka.utils.VerifiableProperties) >.... > >Virendra > >On 7/10/14, 10:07 AM, "Virendra Pratap Singh" ><vpsi...@yahoo-inc.com.INVALID> wrote: > >>Thank you for pointing out to JIRA. >> >>On 7/9/14, 6:38 PM, "Jun Rao" <jun...@gmail.com> wrote: >> >>>This is being worked on in >>>https://issues.apache.org/jira/browse/KAFKA-1325 >>> >>>Thanks, >>> >>>Jun >>> >>> >>>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh < >>>vpsi...@yahoo-inc.com.invalid> wrote: >>> >>>> Well currently the log rollup is controlled via >>>> >>>> log.roll.hours >>>> Or >>>> log.segment.bytes >>>> >>>> Given that we now have support to log retention in minutes, I guess it >>>> would be apt to have rollup also have capability to be available in >>>> minutes. Whom/where should I ask to have that coded in. >>>> >>>> One a similar note, I would genuinely want all the size and time based >>>> parameters to be defined at bytes and ms level. This would make it >>>>generic >>>> enough to have user choose and define kind of setting they want and >>>>not >>>> require someone to go and change the code to support a new use case. >>>> Having hours/minutes in config will just cause proliferation/bloating >>>>of >>>> config parameters. What is a good place to air this suggestion. >>>>Anyone? >>>> >>>> Virendra >>>> >>>> On 7/9/14, 7:21 AM, "Jun Rao" <jun...@gmail.com> wrote: >>>> >>>> >Actually, Kafka only removes old segments. The last (active) segment >>>>is >>>> >never removed. So, f you want to have a 10 min retention, you need to >>>> >configure log rolling such that log segments are rolled at least >>>>every >>>>10 >>>> >mins. >>>> > >>>> >Thanks, >>>> > >>>> >Jun >>>> > >>>> > >>>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh < >>>> >vpsi...@yahoo-inc.com.invalid> wrote: >>>> > >>>> >> That's correct. The server where in I was running 0.8.1.1 was not >>>> >>honoring >>>> >> this parameter, despite the fact it was set in it >>>>server.properties. >>>> >> Not sure if this fact would play any role, the server which was >>>>running >>>> >> 0.8.0 was the leader for all the topics and partition in my setup. >>>>And >>>> >>the >>>> >> second server running 0.8.1.1 has all the replicas (follower). >>>> >> >>>> >> Virendra >>>> >> >>>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: >>>> >> >>>> >> >Server properties should affect on only the local instance >>>>separately. >>>> >>Are >>>> >> >you saying the property is not honored even on the 0.8.1 machines? >>>> >> > >>>> >> >Guozhang >>>> >> > >>>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh < >>>> >> >vpsi...@yahoo-inc.com.invalid> wrote: >>>> >> > >>>> >> >> By setting this property >>>> >> >> log.retention.mins=10 >>>> >> >> in the server.properties file, which is passed as argument when >>>> >>starting >>>> >> >> the broker. >>>> >> >> >>>> >> >> Virendra >>>> >> >> >>>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: >>>> >> >> >>>> >> >> >How do you set the retention.minutes property? Is it through >>>> >>zk-based >>>> >> >> >topics tool? >>>> >> >> > >>>> >> >> >Guozhang >>>> >> >> > >>>> >> >> > >>>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh < >>>> >> >> >vpsi...@yahoo-inc.com.invalid> wrote: >>>> >> >> > >>>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker >>>> >>0.8.0 >>>> >> >>and >>>> >> >> >> the other 0.8.1.1. Should the retention of topics for >>>>partitions >>>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce >>>>the >>>> >>server >>>> >> >> >> properties settings as defined for that server. >>>> >> >> >> >>>> >> >> >> So this brings an interesting question, in case of >>>>heterogeneous >>>> >> >> >> environment (as is in my case, which system parameters will >>>>take >>>> >> >> >> preference/precedence). >>>> >> >> >> >>>> >> >> >> Virendra >>>> >> >> >> >>>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wangg...@gmail.com> >>>>wrote: >>>> >> >> >> >>>> >> >> >> >The retention.minute property is only introduced in 0.8.1: >>>> >> >> >> > >>>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918 >>>> >> >> >> > >>>> >> >> >> >if you are running 0.8.0 then it will not be recognized. >>>> >> >> >> > >>>> >> >> >> >Guozhang >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh < >>>> >> >> >> >vpsi...@yahoo-inc.com.invalid> wrote: >>>> >> >> >> > >>>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the >>>> >>broker1 >>>> >> >>is >>>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache >>>>release >>>> >> >>link. >>>> >> >> >> >> Directly using the tar ball, no local build used). >>>> >> >> >> >> >>>> >> >> >> >> I have set the log.retention.minutes=10. However the >>>>broker >>>>is >>>> >>not >>>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir >>>>at >>>> >>all. >>>> >> >> >> >> >>>> >> >> >> >> However when I set the log.retention.hours=1, then it >>>>starts >>>> >> >>cleaning >>>> >> >> >> >>the >>>> >> >> >> >> log. >>>> >> >> >> >> >>>> >> >> >> >> When I have the log.retention.minutes set in the >>>> >>server.properties >>>> >> >> >>then >>>> >> >> >> >>I >>>> >> >> >> >> see this logged in server.log: >>>> >> >> >> >> >>>> >> >> >> >> Š.. >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property >>>>log.retention.minutes >>>> >>is >>>> >> >>not >>>> >> >> >> >>valid >>>> >> >> >> >> (kafka.utils.VerifiableProperties) >>>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property >>>>log.retention.minutes >>>> >>is >>>> >> >>not >>>> >> >> >> >>valid >>>> >> >> >> >> (kafka.utils.VerifiableProperties) >>>> >> >> >> >> ŠŠ >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> I have set these properties too: >>>> >> >> >> >> >>>> >> >> >> >> log.cleaner.enable=true >>>> >> >> >> >> log.cleanup.policy=delete >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> >> But I see similar warning logged for these properties too. >>>> >> >> >> >> >>>> >> >> >> >> Regards, >>>> >> >> >> >> Virendra >>>> >> >> >> >> >>>> >> >> >> >> >>>> >> >> >> > >>>> >> >> >> > >>>> >> >> >> >-- >>>> >> >> >> >-- Guozhang >>>> >> >> >> >>>> >> >> >> >>>> >> >> > >>>> >> >> > >>>> >> >> >-- >>>> >> >> >-- Guozhang >>>> >> >> >>>> >> >> >>>> >> > >>>> >> > >>>> >> >-- >>>> >> >-- Guozhang >>>> >> >>>> >> >>>> >>>> >> >
