Well currently the log rollup is controlled via log.roll.hours Or log.segment.bytes
Given that we now have support to log retention in minutes, I guess it would be apt to have rollup also have capability to be available in minutes. Whom/where should I ask to have that coded in. One a similar note, I would genuinely want all the size and time based parameters to be defined at bytes and ms level. This would make it generic enough to have user choose and define kind of setting they want and not require someone to go and change the code to support a new use case. Having hours/minutes in config will just cause proliferation/bloating of config parameters. What is a good place to air this suggestion. Anyone? Virendra On 7/9/14, 7:21 AM, "Jun Rao" <jun...@gmail.com> wrote: >Actually, Kafka only removes old segments. The last (active) segment is >never removed. So, f you want to have a 10 min retention, you need to >configure log rolling such that log segments are rolled at least every 10 >mins. > >Thanks, > >Jun > > >On Tue, Jul 8, 2014 at 10:04 PM, Virendra Pratap Singh < >vpsi...@yahoo-inc.com.invalid> wrote: > >> That's correct. The server where in I was running 0.8.1.1 was not >>honoring >> this parameter, despite the fact it was set in it server.properties. >> Not sure if this fact would play any role, the server which was running >> 0.8.0 was the leader for all the topics and partition in my setup. And >>the >> second server running 0.8.1.1 has all the replicas (follower). >> >> Virendra >> >> On 7/8/14, 12:54 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: >> >> >Server properties should affect on only the local instance separately. >>Are >> >you saying the property is not honored even on the 0.8.1 machines? >> > >> >Guozhang >> > >> >On Mon, Jul 7, 2014 at 3:55 PM, Virendra Pratap Singh < >> >vpsi...@yahoo-inc.com.invalid> wrote: >> > >> >> By setting this property >> >> log.retention.mins=10 >> >> in the server.properties file, which is passed as argument when >>starting >> >> the broker. >> >> >> >> Virendra >> >> >> >> On 7/7/14, 3:31 PM, "Guozhang Wang" <wangg...@gmail.com> wrote: >> >> >> >> >How do you set the retention.minutes property? Is it through >>zk-based >> >> >topics tool? >> >> > >> >> >Guozhang >> >> > >> >> > >> >> >On Mon, Jul 7, 2014 at 3:07 PM, Virendra Pratap Singh < >> >> >vpsi...@yahoo-inc.com.invalid> wrote: >> >> > >> >> >> I am running a mixed cluster as I mentioned earlier. 1 broker >>0.8.0 >> >>and >> >> >> the other 0.8.1.1. Should the retention of topics for partitions >> >> >> owned/replicated by the broker running 0.8.1.1 not enforce the >>server >> >> >> properties settings as defined for that server. >> >> >> >> >> >> So this brings an interesting question, in case of heterogeneous >> >> >> environment (as is in my case, which system parameters will take >> >> >> preference/precedence). >> >> >> >> >> >> Virendra >> >> >> >> >> >> On 6/30/14, 9:19 AM, "Guozhang Wang" <wangg...@gmail.com> wrote: >> >> >> >> >> >> >The retention.minute property is only introduced in 0.8.1: >> >> >> > >> >> >> >https://issues.apache.org/jira/browse/KAFKA-918 >> >> >> > >> >> >> >if you are running 0.8.0 then it will not be recognized. >> >> >> > >> >> >> >Guozhang >> >> >> > >> >> >> > >> >> >> > >> >> >> >On Fri, Jun 27, 2014 at 2:13 PM, Virendra Pratap Singh < >> >> >> >vpsi...@yahoo-inc.com.invalid> wrote: >> >> >> > >> >> >> >> Running a mixed 2 broker cluster. Mixed as in one of the >>broker1 >> >>is >> >> >> >> running 0.8.0 and broker2 one 0.8.1.1 (from the apache release >> >>link. >> >> >> >> Directly using the tar ball, no local build used). >> >> >> >> >> >> >> >> I have set the log.retention.minutes=10. However the broker is >>not >> >> >> >> honoring the setting. I see its not cleaning the log.dir at >>all. >> >> >> >> >> >> >> >> However when I set the log.retention.hours=1, then it starts >> >>cleaning >> >> >> >>the >> >> >> >> log. >> >> >> >> >> >> >> >> When I have the log.retention.minutes set in the >>server.properties >> >> >>then >> >> >> >>I >> >> >> >> see this logged in server.log: >> >> >> >> >> >> >> >> Š.. >> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes >>is >> >>not >> >> >> >>valid >> >> >> >> (kafka.utils.VerifiableProperties) >> >> >> >> [2014-06-27 19:21:06,633] WARN Property log.retention.minutes >>is >> >>not >> >> >> >>valid >> >> >> >> (kafka.utils.VerifiableProperties) >> >> >> >> ŠŠ >> >> >> >> >> >> >> >> >> >> >> >> I have set these properties too: >> >> >> >> >> >> >> >> log.cleaner.enable=true >> >> >> >> log.cleanup.policy=delete >> >> >> >> >> >> >> >> >> >> >> >> But I see similar warning logged for these properties too. >> >> >> >> >> >> >> >> Regards, >> >> >> >> Virendra >> >> >> >> >> >> >> >> >> >> >> > >> >> >> > >> >> >> >-- >> >> >> >-- Guozhang >> >> >> >> >> >> >> >> > >> >> > >> >> >-- >> >> >-- Guozhang >> >> >> >> >> > >> > >> >-- >> >-- Guozhang >> >>
