Jay made a good point. The docs says it is for the topic but actually it is for one partition. Thanks, Jun.
Regards, Libo -----Original Message----- From: Jun Rao [mailto:jun...@gmail.com] Sent: Wednesday, August 28, 2013 11:02 AM To: users@kafka.apache.org Subject: Re: questtion about log.retention.bytes Changed the wording in the doc. Thanks, Jun On Wed, Aug 28, 2013 at 7:20 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > I think the problem is that there is no way to understand the meaning > of that config from the docs, so people keep asking over and over > again. The docs make iit sound like it is per topic and just says it > is the "maximum size before it is deleted" which makes no sense... > > -jay > > On Tuesday, August 27, 2013, Jun Rao wrote: > > > For the first question, yes. > > > > For the second one, this is documented in > > http://kafka.apache.org/documentation.html#brokerconfigs > > > > "Note that all per topic configuration properties below have the > > format > of > > csv (e.g., "topic1:value1,topic2:value2")." > > Thanks, > > Jun > > > > > > > > On Tue, Aug 27, 2013 at 11:52 AM, Yu, Libo <libo...@citi.com > <javascript:;>> > > wrote: > > > > > Hi Jun, > > > > > > In a previous email thread > > > > > > http://markmail.org/search/?q=kafka+log.retention.bytes#query:kafka%20 > log.retention.bytes+page:1+mid:qnt4pbq47goii2ui+state:results > > > , > > > you said log.retention.bytes is for each partition. Could you > > > clarify > on > > > that? > > > > > > Say if I have a topic with three partitions. I want to limit the > > > disk space to 1Gb for each partition. > > > Then log.retention.bytes should be set to 1Gb (not 3Gb). Is that right? > > > > > > If I want to use log.retention.bytes.per.topic to set the same > > > limit, should it be set to 1G or 3G? > > > > > > Libo > > > > > > > > > -----Original Message----- > > > From: Jun Rao [mailto:jun...@gmail.com <javascript:;>] > > > Sent: Saturday, August 17, 2013 12:40 AM > > > To: users@kafka.apache.org <javascript:;> > > > Subject: Re: questtion about log.retention.bytes > > > > > > log.retention.bytes is for all topics that are not included in > > > log.retention.bytes.per.topic (which defines a map of topic -> size). > > > > > > Currently, we don't have a total size limit across all topics. > > > > > > Thanks, > > > > > > Jun > > > > > > > > > On Fri, Aug 16, 2013 at 2:00 PM, Paul Christian > > > <pchrist...@salesforce.com <javascript:;>>wrote: > > > > > > > According to the Kafka 8 documentation under broker configuration. > > > > There are these parameters and their definitions. > > > > > > > > log.retention.bytes -1 The maximum size of the log before > > > > deleting it log.retention.bytes.per.topic "" The maximum size of > > > > the log for some specific topic before deleting it > > > > > > > > I'm curious what the first value 'log.retention.bytes' is for if > > > > the second one is for per topic logs, because aren't all logs > > > > generated per topic? Is this an aggregate value across topics? > > > > > > > > Related question, is there a parameter for kafka where you can > > > > say only hold this much TOTAL data across all topic ( logs/index > > > > together > > )? > > > I.e. > > > > our hosts have this much available space and so value > > > > log.retention.whatever.aggregate == 75% total disk space. > > > > > > > > > >
