log(disable by default), so
> this is motivation for this KIP.
>
>
> Thanks,
> David
>
>
>
>
>
>
>
> -- ???? --------------
> ??????: "Becket Qin";;
> : 2016??11??6??(??) 11:39
> ??: "dev";
>
> : Re: [DI
d
>
>
>
>
>
>
>
> -- 原始邮件 --------------
> 发件人: "Becket Qin";;
> 发送时间: 2016年11月6日(星期天) 晚上11:39
> 收件人: "dev";
>
> 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
>
>
>
> Hi David,
>
sable by default), so this is
motivation for this KIP.
Thanks,
David
-- --
??: "Becket Qin";;
: 2016??11??6??(??) 11:39
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log ret
he KIP
> > mentioned. This method is simple for the users and it can automatically
> > clean the consumed log, but it will add more query work to the brokers.
> >
> >
> > Which method is better?
> >
> >
> > Thanks,
> > David
> >
> &
with kafka all the time, this
may not always holds.
Thanks,
David
-- --
??: "Becket Qin";;
: 2016??11??1??(??) 3:57
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
gt; clean the consumed log, but it will add more query work to the brokers.
>
>
> Which method is better?
>
>
> Thanks,
> David
>
>
>
>
>
>
>
>
> ------ 原始邮件 --------------
> 发件人: "Mayuresh Gharat";;
> 发送时间: 2016年10月29
: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
I do agree with Guozhang on having applications request an external
service(admin) that talks to kafka, for trimming, in which case this
external service(admin) can check if its OK to send the t
ntics is exactly the same as to KIP-47, while the only
> > > > > difference
> > > > > > > is
> > > > > > > > > that we use offset instead of timestamp to indicate, which
> > can
> > > be
> > > > > > honor
> > > > > > > > by
> > > > > > > > > the same
gt; > > > > > > does the min() calculation, rather only keeping / deleting
> > > messages
> > > > > > from
> > > > > > > > client admin requests, and 2) allowing more ge
gt; > > >
> > > > > > >
> > > > > > >
> > > > > > > Guozhang
> > > > > > >
> > > > > > >
> > > > > > > On Sat, Oct 22, 2016 at 5:46 PM, Becket Qin <
> > becket@gmail.com>
> >
t; > > > DAG,
> > > > > > > there will be a bunch of intermediate result, we only care
> about
> > > the
> > > > > > > consumer group that is in the downstream of the DAG, but not
> > other
> > > > > > groups.
>
; > > > > > all the downstream processing jobs has successfully processed the
> > > > > messages.
> > > > > > In that case, we only care about the downstream processing jobs,
> > but
> > > > not
> > > > > > other groups. That means if a down stream job did
; > >
> > > > >
> > > > > 2. Yes, the configuration should be at topic level and set
> > dynamically.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > &g
mmitted
> > > > > offset
> > > > > > based log retention.
> > > > > >
> > > > > > It seems the current proposal does not have an "interested
> consumer
> > > > group
> > > > > > set" configur
t; > >
> > > >
> > > > 2. If the console consumer reading and commit, its commit offset
> will
> > be
> > > > used to calculate the min commit offset for this topic.
> > > > We can avoid the random consumer using the method Becket suggested.
&
ing and commit, its commit offset will
> be
> > > used to calculate the min commit offset for this topic.
> > > We can avoid the random consumer using the method Becket suggested.
> > >
> > >
> > > 3. It will not delete the log immediately, the log wi
are less than the min commit offset. So
> > the user can rewind its offset in the log.retention.ms.
> >
> >
> > Thanks,
> > David
> >
> >
> >
> >
> > -- 原始邮件 --
> > 发件人: "Mayuresh Gharat";;
>
gt;
>
> -- 原始邮件 --
> 发件人: "Mayuresh Gharat";;
> 发送时间: 2016年10月19日(星期三) 上午10:25
> 收件人: "dev";
>
> 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
>
>
>
> Hi David,
>
> Thanks for
harat";;
: 2016??10??19??(??) 10:25
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
Hi David,
Thanks for the KIP.
I had some questions/suggestions :
It would be great if you can explain with an example about how the mi
too, the
specified consumed group may not consuming all the topics in this brokers.
Thanks,
David
-- --
??: "Becket Qin";;
: 2016??10??17??(??) 3:27
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed
it offset is
> > also in the __commit_offsets topic and
> > stay in the offset cache, we can delete it via this API.
> >
> >
> > Thanks,
> > David
> >
> >
> > -- 原始邮件 --
> > 发件人: "Dong Lin";;
>
roup is in inactive, but it's commit offset is
> also in the __commit_offsets topic and
> stay in the offset cache, we can delete it via this API.
>
>
> Thanks,
> David
>
>
> -- 原始邮件 ----------
> 发件人: "Dong Lin";;
> 发送时间: 2016年10月
David
-- --
??: "Dong Lin";;
: 2016??10??14??(??) 5:01
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
Hi David,
As explained in the motivation section of the KIP, the problem is that if
log retention is to
h are
consumed. And after 7 days, even if the messages are not
consumed, we will clean it anyway.
-- --
??: "Dong Lin";;
: 2016??10??14??(??) 1:23
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log re
ev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
Hi David
This is a very timely KIP given the number of use cases in the streams
processing pipeline than need consumed log retention management.
Some questions that Becket and Dong asked just wanted to make
re consistent with the
> purpose of having this forced log retention (to save disk space). And if we
> adopt the 2nd solution, the use-case you suggested can be easily addressed
> by setting forced log retention to 1 day and enable consumed log retention.
> What do you think?
>
>
>
&
avid
>
>
>
>
> -- 原始邮件 --------------
> 发件人: "Dong Lin";;
> 发送时间: 2016年10月10日(星期一) 下午4:05
> 收件人: "dev";
>
> 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
>
>
>
> Hey David,
>
>
g Lin";;
: 2016??10??10??(??) 4:05
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
Hey David,
Thanks for reply. Please see comment inline.
On Mon, Oct 10, 2016 at 12:40 AM, Pengwei (L) wrote:
> Hi Dong
>Thanks for the quest
Hi David
This is a very timely KIP given the number of use cases in the streams
processing pipeline than need consumed log retention management.
Some questions that Becket and Dong asked just wanted to make sure are
described in the KIP.
1. How is the configuration setup per topic to know what
Hey David,
Thanks for reply. Please see comment inline.
On Mon, Oct 10, 2016 at 12:40 AM, Pengwei (L) wrote:
> Hi Dong
>Thanks for the questions:
>
> 1. Now we don't distinguish inactive or active groups. Because in some
> case maybe inactive group will become active again, and using the p
Hi Dong
Thanks for the questions:
1. Now we don't distinguish inactive or active groups. Because in some case
maybe inactive group will become active again, and using the previous commit
offset.
So we will not delete the log segment in the consumer retention if there are
some groups consum
ttle different from this
> KIP, though we are all modifying the log retention.
>
>
> Thanks,
> David.
>
>
>
>
>
>
>
>
> ---------- 原始邮件 ----------
> 发件人: "Becket Qin";;
> 发送时间: 2016年10月9日(星期天) 中午1:00
> 收件人: "dev";
>
on.
Thanks,
David.
-- --
??: "Becket Qin";;
: 2016??10??9??(??) 1:00
??: "dev";
????: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
Hi David,
Thanks for the explanation. Could you update the KIP-68 wiki to include the
Hey David,
Thanks for the KIP. Can you help with the following two questions:
1) If someone start a consumer (e.g. kafka-console-consumer) to consume a
topic for debug/validation purpose, a randome consumer group may be created
and offset may be committed for this consumer group. If no offset com
Hi David,
Thanks for the explanation. Could you update the KIP-68 wiki to include the
changes that need to be made?
I have a few more comments below:
1. We already have an internal topic __consumer_offsets to store all the
committed offsets. So the brokers can probably just consume from that to
Hi Becket,
Thanks for the feedback:
1. We use the simple consumer api to query the commit offset, so we don't need
to specify the consumer group.
2. Every broker using the simple consumer api(OffsetFetchKey) to query the
commit offset in the log retention process. The client can commit offs
Hi Pengwei,
Thanks for the KIP proposal. It is a very useful KIP. At a high level, the
proposed behavior looks reasonable to me.
However, it seems that some of the details are not mentioned in the KIP.
For example,
1. How will the expected consumer group be specified? Is it through a per
topic d
Hi All,
I have made a KIP to enhance the log retention, details as follows:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-68+Add+a+consumed+log+retention+before+log+retention
Now start a discuss thread for this KIP , looking forward to the feedback.
Thanks,
David
38 matches
Mail list logo
