Has someone really used : log.retention.minutes in kafka 0.8.1.1. I have my full cluster running on 0.8.1.1 and the logs data is just not getting cleaned up.
And I see this message in kafka server.log ... [2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid (kafka.utils.VerifiableProperties) [2014-07-10 20:49:43,786] WARN Property log.retention.minutes is not valid (kafka.utils.VerifiableProperties) .... Virendra On 7/10/14, 10:07 AM, "Virendra Pratap Singh" <vpsi...@yahoo-inc.com.INVALID> wrote: >Thank you for pointing out to JIRA. > >On 7/9/14, 6:38 PM, "Jun Rao" <jun...@gmail.com> wrote: > >>This is being worked on in >>https://issues.apache.org/jira/browse/KAFKA-1325 >> >>Thanks, >> >>Jun >> >> >>On Wed, Jul 9, 2014 at 11:42 AM, Virendra Pratap Singh < >>vpsi...@yahoo-inc.com.invalid> wrote: >> >>> Well currently the log rollup is controlled via >>> >>> log.roll.hours >>> Or >>> log.segment.bytes >>> >>> Given that we now have support to log retention in minutes, I guess it >>> would be apt to have rollup also have capability to be available in >>> minutes. Whom/where should I ask to have that coded in. >>> >>> One a similar note, I would genuinely want all the size and time based >>> parameters to be defined at bytes and ms level. This would make it >>>generic >>> enough to have user choose and define kind of setting they want and not >>> require someone to go and change the code to support a new use case. >>> Having hours/minutes in config will just cause proliferation/bloating >>>of >>> config parameters. What is a good place to air this suggestion. Anyone? >>> >>> Virendra >>> >>> On 7/9/14, 7:21 AM, "Jun Rao" <jun...@gmail.com> wrote: >>> >>> >Actually, Kafka only removes old segments. The last (active) segment >>>is >>> >never removed. So, f you want to have a 10 min retention, you need to >>> >configure log rolling such that log segments are rolled at least every >>>10 >>> >mins. >>> > >>> >Thanks, >>> > >>> >Jun >>> > >>> > >>> >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh < >>> >vpsi...@yahoo-inc.com.invalid> wrote: >>> > >>> >> That's correct. The server where in I was running 0.8.1.1 was not >>> >>honoring >>> >> this parameter, despite the fact it was set in it server.properties. >>> >> Not sure if this fact would play any role, the server which was >>>running >>> >> 0.8.0 was the leader for all the topics and partition in my setup. >>>And >>> >>the >>> >> second server running 0.8.1.1 has all the replicas (follower). >>> >> >>> >> Virendra >>> >> >>> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: >>> >> >>> >> >Server properties should affect on only the local instance >>>separately. >>> >>Are >>> >> >you saying the property is not honored even on the 0.8.1 machines? >>> >> > >>> >> >Guozhang >>> >> > >>> >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh < >>> >> >vpsi...@yahoo-inc.com.invalid> wrote: >>> >> > >>> >> >> By setting this property >>> >> >> log.retention.mins=10 >>> >> >> in the server.properties file, which is passed as argument when >>> >>starting >>> >> >> the broker. >>> >> >> >>> >> >> Virendra >>> >> >> >>> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: >>> >> >> >>> >> >> >How do you set the retention.minutes property? Is it through >>> >>zk-based >>> >> >> >topics tool? >>> >> >> > >>> >> >> >Guozhang >>> >> >> > >>> >> >> > >>> >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh < >>> >> >> >vpsi...@yahoo-inc.com.invalid> wrote: >>> >> >> > >>> >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker >>> >>0.8.0 >>> >> >>and >>> >> >> >> the other 0.8.1.1. Should the retention of topics for >>>partitions >>> >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce the >>> >>server >>> >> >> >> properties settings as defined for that server. >>> >> >> >> >>> >> >> >> So this brings an interesting question, in case of >>>heterogeneous >>> >> >> >> environment (as is in my case, which system parameters will >>>take >>> >> >> >> preference/precedence). >>> >> >> >> >>> >> >> >> Virendra >>> >> >> >> >>> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wangg...@gmail.com> >>>wrote: >>> >> >> >> >>> >> >> >> >The retention.minute property is only introduced in 0.8.1: >>> >> >> >> > >>> >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918 >>> >> >> >> > >>> >> >> >> >if you are running 0.8.0 then it will not be recognized. >>> >> >> >> > >>> >> >> >> >Guozhang >>> >> >> >> > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh < >>> >> >> >> >vpsi...@yahoo-inc.com.invalid> wrote: >>> >> >> >> > >>> >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the >>> >>broker1 >>> >> >>is >>> >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache >>>release >>> >> >>link. >>> >> >> >> >> Directly using the tar ball, no local build used). >>> >> >> >> >> >>> >> >> >> >> I have set the log.retention.minutes=10. However the broker >>>is >>> >>not >>> >> >> >> >> honoring the setting. I see its not cleaning the log.dir at >>> >>all. >>> >> >> >> >> >>> >> >> >> >> However when I set the log.retention.hours=1, then it >>>starts >>> >> >>cleaning >>> >> >> >> >>the >>> >> >> >> >> log. >>> >> >> >> >> >>> >> >> >> >> When I have the log.retention.minutes set in the >>> >>server.properties >>> >> >> >>then >>> >> >> >> >>I >>> >> >> >> >> see this logged in server.log: >>> >> >> >> >> >>> >> >> >> >> Š.. >>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property >>>log.retention.minutes >>> >>is >>> >> >>not >>> >> >> >> >>valid >>> >> >> >> >> (kafka.utils.VerifiableProperties) >>> >> >> >> >> [2014-06-27 19:21:06,633] WARN Property >>>log.retention.minutes >>> >>is >>> >> >>not >>> >> >> >> >>valid >>> >> >> >> >> (kafka.utils.VerifiableProperties) >>> >> >> >> >> ŠŠ >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> I have set these properties too: >>> >> >> >> >> >>> >> >> >> >> log.cleaner.enable=true >>> >> >> >> >> log.cleanup.policy=delete >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> >> But I see similar warning logged for these properties too. >>> >> >> >> >> >>> >> >> >> >> Regards, >>> >> >> >> >> Virendra >>> >> >> >> >> >>> >> >> >> >> >>> >> >> >> > >>> >> >> >> > >>> >> >> >> >-- >>> >> >> >> >-- Guozhang >>> >> >> >> >>> >> >> >> >>> >> >> > >>> >> >> > >>> >> >> >-- >>> >> >> >-- Guozhang >>> >> >> >>> >> >> >>> >> > >>> >> > >>> >> >-- >>> >> >-- Guozhang >>> >> >>> >> >>> >>> >
