Re: [VOTE] KIP-79 - ListOffsetRequest v1 and search by timestamp methods in new consumer.

Thanks everyone for the votes, I want to mention a minor change made to the interface. To ensure the accordance with KIP-45, we have changed the "earliestOffset()" and "latestOffset()" method to take Collection instead of Set. Thanks, Jiangjie (Becket) Qin On Sun, Sep

Re: [VOTE] KIP-79 - ListOffsetRequest v1 and search by timestamp methods in new consumer.

apis, would it be better to rename > > earliestOffsets() and latestOffsets() to beginningOffsets() and > > endOffsets()? > Good point, will make the change. This makes the name align with seekToBeginning() and seekToEnd(). > > > Jun > > > > On Fri,

Re: [VOTE] KIP-79 - ListOffsetRequest v1 and search by timestamp methods in new consumer.

016 at 12:38 AM, Becket Qin wrote: > > > Hi all, > > > > I'd like to start the voting for KIP-79 > > > > In short we propose to : > > 1. add a ListOffsetRequest/ListOffsetResponse v1, and > > 2. add earliestOffsts(), latestOffsets() and offsetForTime

Re: [UPDATE] 0.10.1 Release Progress

Awesome! On Mon, Sep 19, 2016 at 11:42 PM, Neha Narkhede wrote: > Nice! > On Mon, Sep 19, 2016 at 11:33 PM Ismael Juma wrote: > > > Well done everyone. :) > > > > On 20 Sep 2016 5:23 am, "Jason Gustafson" wrote: > > > > > Thanks everyone for the hard work! The 0.10.1 release branch has been >

Re: Does Kafka 0.9 can guarantee not loss data

In order to satisfy a produce response, there are two conditions: A. The leader's high watermark should be higher than the requiredOffset (max offset in that produce request of that partition) B. The number of in sync replica is greater than min.isr. The ultimate goal here is to make sure at least

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

consumers directly consuming from it. 4. Is there any other changes need to be made (e.g. new protocols) in addition to the configuration change? It would be great if you can update the wiki to have more details. Thanks, Jiangjie (Becket) Qin On Wed, Sep 7, 2016 at 2:26 AM, Pengwei (L) wrote

Re: [DISCUSS] KIP-82 - Add Record Headers

Yes, KIP-74 has already been checked in. The new FetchRequest/Response version should be V4. :) On Mon, Oct 3, 2016 at 10:14 AM, Sean McCauliff < smccaul...@linkedin.com.invalid> wrote: > Change to public interfaces: > > "Add ProduceRequest/ProduceResponse V3 which uses the new message format. >

Re: [UPDATE] 0.10.1 Release Progress

gt; > > > > and transient test failures. If you are looking to help out, > > addressing > > > > the > > > > > transient test failures are the biggest need since I'm sure you've > > > > noticed > > &g

Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy

has reached beyond log.retention.bytes but the timestamp has not reached log.retention.min.timestamp, do we delete the segment? Thanks, Jiangjie (Becket) Qin On Fri, Jun 3, 2016 at 11:02 AM, Magnus Edenhill wrote: > Bumping this thread so Wes can reply to it. Ignore this mail. > > 2016-0

Re: client use high cpu which caused by delayedFetch operation immediately return

Can you check if you have KAFKA-3003 when you run the code? On Sat, Oct 8, 2016 at 12:52 AM, Kafka wrote: > Hi all, > we found our consumer have high cpu load in our product > enviroment，as we know，fetch.min.bytes and fetch.wait.ma < > http://fetch.wait.ma/>x.ms will affect the frequency

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

propose a new configuration on log retention. It may be worth thinking about the behavior together. Thanks, Jiangjie (Becket) Qin On Sat, Oct 8, 2016 at 2:15 AM, Pengwei (L) wrote: > Hi Becket, > > Thanks for the feedback: > 1. We use the simple consumer api to query the commit o

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

above, it seems that we need some way to configure the set of consumer groups a topic should be listening on? If it is through topic config, it would be good to document the configuration name and format of value in the wiki as well. Thanks, Jiangjie (Becket) Qin On Sun, Oct 9, 2016 at 7:14 AM

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

Or we can use the intermediate topic for some debugging work. Thanks, Jiangjie (Becket) Qin On Sun, Oct 16, 2016 at 2:15 AM, 东方甲乙 <254479...@qq.com> wrote: > Hi Dong, > The KIP is used to solve both these 2 cases, we specify a small > consumed log retention time to deleted the

Re: client use high cpu which caused by delayedFetch operation immediately return

Glad to know :) On Tue, Oct 18, 2016 at 1:24 AM, Json Tu wrote: > Thanks. I patch it, and everything goes ok. > > 在 2016年10月9日，下午12:39，Becket Qin 写道： > > > > Can you check if you have KAFKA-3003 when you run the code? > > > > On Sat, Oct 8, 2016 at 12:52

Re: [ANNOUNCE] Apache Kafka 0.10.1.0 Released

Congratulations! Great job, Jason. On Thu, Oct 20, 2016 at 11:57 AM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > +1 > > Thanks Jason for driving the release. Great job! > > --Vahid > > > > > From: Sriram Subramanian > To: dev@kafka.apache.org > Date: 10/20/2016 11:45 AM > Sub

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

to wait for that job. Without the predefined interested group, it is hard to achieve this. 2. Yes, the configuration should be at topic level and set dynamically. Thanks, Jiangjie (Becket) Qin On Fri, Oct 21, 2016 at 7:40 AM, 东方甲乙 <254479...@qq.com> wrote: > Hi Mayuresh, > Thank

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

this issue. But I am not sure if that the requirement of separating offset commit from consume-based log retention is general enough to be addressed specifically. Thanks, Jiangjie (Becket) Qin On Tue, Oct 25, 2016 at 3:00 PM, Joel Koshy wrote: > +1 - I was thinking the exact same thing. >

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

oblem. The use cases for multiple consumer groups I am thinking of is some kind of fork in the DAG, i.e. one intermediate result stream used by multiple downstream jobs. But that may not be a big deal if the processing is within the same application. Thanks, Jiangjie (Becket) Qin On Tue, Oct 25

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a > > committer and we are pleased to announce that he has accepted! > > > > Becket has made significant contributions to Kafka over the last two > years. > > He has been deeply involved in a broad range of KI

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

you take a look and see if that could be used for KIP-68 as well? Thanks, Jiangjie (Becket) Qin On Sun, Oct 30, 2016 at 2:24 AM, 东方甲乙 <254479...@qq.com> wrote: > Hi All, > > > As per our discussion, there are two ways to clean the consumed log: > > > 1) Use an

Re: Kafka client circuit breaker

Hi Andrey, I am not sure I understand the question "why doesn't Kafka client have the solution for this kind of problems?" Can you elaborate a little on what kind of problems are you referring to? And what do you mean by "circuit breaker" here? Jiangjie (Becket) Qin O

Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag

case. It also help avoid the potential ambiguity. maybe it is better to bump up the magic value. Thanks, Jiangjie (Becket) Qin On Mon, Oct 31, 2016 at 4:17 PM, Jay Kreps wrote: > Xavier, > > Yeah I think that post KIP-58 it is possible to depend on the delivery of > messages

Re: Add a customized logo for Kafka Streams

I also like this one! Jiangjie (Becket) Qin On Fri, Mar 13, 2020 at 9:07 AM Matthias J. Sax wrote: > I personally love it! > > -Matthias > > On 3/12/20 11:31 AM, Sophie Blee-Goldman wrote: > > How reasonable of it. Let's try this again: > > Strea

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congratulations! Rajini! Great work! On Mon, Apr 24, 2017 at 3:33 PM, Jason Gustafson wrote: > Woohoo! Great work, Rajini! > > On Mon, Apr 24, 2017 at 3:27 PM, Jun Rao wrote: > > > Congratulations, Rajini ! Thanks for all your contributions. > > > > Jun > > > > On Mon, Apr 24, 2017 at 2:06 PM,

[VOTE] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

fine. Thanks, Jiangjie (Becket) Qin

Re: [VOTE] KIP-126 - Allow KafkaProducer to batch based on uncompressed size

I just updated the KIP name to reflect the actual proposal. I'll close this vote and start another one to avoid confusion. On Tue, Apr 25, 2017 at 12:27 PM, Dong Lin wrote: > +1 (non-binding) > > On Tue, Apr 25, 2017 at 10:42 AM, Becket Qin wrote: > > > Hi, > >

[VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

ime in our mirror maker and it looks working fine. Thanks, Jiangjie (Becket) Qin

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

Bump. On Tue, Apr 25, 2017 at 3:17 PM, Bill Bejeck wrote: > +1 > > On Tue, Apr 25, 2017 at 4:43 PM, Dong Lin wrote: > > > +1 (non-binding) > > > > On Tue, Apr 25, 2017 at 12:33 PM, Becket Qin > wrote: > > > > > Hi, > > > > > >

Streams Processing Meetup @LinkedIn

Hello! LinkedIn will be hosting a Streams Processing meetup on Wednesday May 24, 6-9PM in our Sunnyvale HQ. We'll have 3 exciting talks planned: - Streaming Data Pipelines with Brooklin - Kafka at Half the Price

Re: [VOTE] KIP-126 - Allow KafkaProducer to split and resend oversized batches.

Thanks everyone for voting. KIP-126 has passed with +3 binding (Joel, Jason and me) and +3 non-binding (Dong, Bill, Michael) Thanks, Jiangjie (Becket) Qin On Mon, May 8, 2017 at 10:06 AM, Jason Gustafson wrote: > +1 > > On Thu, May 4, 2017 at 9:11 PM, Michael Pearce > wro

Re: [DISCUSS] KIP-143: Controller Health Metrics

keeps happening, we should see a long state propagation time reported by the controller or a long LeaderAndIsrRequest queue time. Thanks, Jiangjie (Becket) Qin On Mon, May 8, 2017 at 2:47 PM, Joel Koshy wrote: > Hi Ismael, > > > > > What about a broker that is not the contr

Re: [DISCUSS] KIP-143: Controller Health Metrics

. Among the above metrics, 4,5,6 could potentially be per broker. Thanks, Jiangjie (Becket) Qin On Mon, May 8, 2017 at 7:24 PM, Onur Karaman wrote: > I had a similar comment to Becket but accidentally posted it on the vote > thread last Friday. From that thread: > "I noticed

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

+1 On Tue, May 9, 2017 at 3:06 AM, Rajini Sivaram wrote: > Hi Jun, Thank you for the KIP. > > +1 (binding) > > On Mon, May 8, 2017 at 6:23 PM, Guozhang Wang wrote: > > > +1 > > > > On Mon, May 8, 2017 at 2:25 PM, Joel Koshy wrote: > > > > > +1 > > > > > > On Mon, May 8, 2017 at 8:30 AM, Roger

Re: [DISCUSS] KIP-143: Controller Health Metrics

Hi Ismael, Yes, a follow up KIP after the controller code settles down sounds good. Thanks, Jiangjie (Becket) Qin On Wed, May 10, 2017 at 6:11 PM, Ismael Juma wrote: > Thanks Jun. Discussed this offline with Onur and Jun and I believe there's > agreement so updated the KIP

Re: [VOTE] KIP-113 - Support replicas movement between log directories

+1 Thanks for the KIP, Dong. On Tue, Jun 6, 2017 at 10:20 AM, Jun Rao wrote: > Hi, Dong, > > Thanks for the KIP. +1 from me. > > Jun > > On Mon, Apr 3, 2017 at 9:49 AM, Dong Lin wrote: > > > Hi all, > > > > It seems that there is no further concern with the KIP-113. We would like > > to start

Re: 0.9.0 release branch

Hi Jun, I added KAFKA-2722 as a blocker for 0.9. It fixes the ISR propagation scalability issue we saw. Thanks, Jiangjie (Becket) Qin On Mon, Nov 2, 2015 at 9:16 AM, Jun Rao wrote: > Hi, everyone, > > We are getting close to the 0.9.0 release. The current plan is to have the &g

Re: [DISCUSS] KIP-32 - Add CreateTime and LogAppendTime to Kafka message

Any thoughts? Thanks, Jiangjie (Becket) Qin On Mon, Dec 7, 2015 at 10:33 AM, Jiangjie Qin wrote: > Bump up this thread. > > Just to recap, the last proposal Jay made (with some implementation details > added) was: > >1. Allow user to stamp the message when produce >

Question about ZookeeperConsumerConnector

.scala:185) at kafka.tools.MirrorMaker$$anonfun$cleanShutdown$1.apply(MirrorMaker.scala:185) at scala.collection.immutable.List.foreach(List.scala:318) at kafka.tools.MirrorMaker$.cleanShutdown(MirrorMaker.scala:185) at kafka.tools.MirrorMaker$$anon$1.run(MirrorMaker.scala:169) Tha

<    1   2   3   4