Re: Replica Fetcher Reset Its Offset to beginning

It might be related to KAFKA-2477. On Thu, Oct 29, 2015 at 6:44 AM, Andrew Otto wrote: > Hi all, > > This morning I woke up to see a very high max replica lag on one of my > brokers. I looked at logs, and it seems that one of the replica fetchers > for a partition just decided that its offset w

Re: 0.9.0 release branch

Hi Jun, I added KAFKA-2722 as a blocker for 0.9. It fixes the ISR propagation scalability issue we saw. Thanks, Jiangjie (Becket) Qin On Mon, Nov 2, 2015 at 9:16 AM, Jun Rao wrote: > Hi, everyone, > > We are getting close to the 0.9.0 release. The current plan is to have the &g

Re: High delay during controlled shutdown and acks=-1

s not very efficient. We might improve it and hopefully later on it won't slow down the replication on the shutting down broker. Thanks, Jiangjie (Becket) Qin On Mon, Nov 2, 2015 at 5:52 AM, Federico Giraud wrote: > Hi, > > I have few java async producers sending data to a 4-node Kafka

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

sonally I think the current interface handles both single partition and a list of partitions better. It is not ideal that we have to adapt to the interface. I just feel it is weirder to create a new list. Thanks, Jiangjie (Becket) Qin On Wed, Jan 27, 2016 at 2:24 PM, Gwen Shapira wrote: > I

Re: Kafka 0.9.0.1 plan

, Jiangjie (Becket) Qin On Fri, Feb 5, 2016 at 10:28 AM, Jun Rao wrote: > Hi, Everyone, > > We have fixed a few critical bugs since 0.9.0.0 was released and are still > investigating a few more issues. The current list of issues tracked for > 0.9.0.1 can be found below. Among them,

Re: [DISCUSS] KIP-45 Standardize all client sequence interaction on j.u.Collection.

Hi Jason, Yes, 0.9 clients should still work with 0.10 brokers. Thanks, Jiangjie (Becket) Qin On Mon, Mar 7, 2016 at 4:10 PM, Jason Gustafson wrote: > +users > > On Mon, Mar 7, 2016 at 4:09 PM, Jason Gustafson > wrote: > > > Hey Ismael, > > > > Thanks f

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats, Onur! On Mon, Nov 6, 2017 at 2:45 PM, James Cheng wrote: > Congrats Onur! Well deserved! > > -James > > > On Nov 6, 2017, at 9:24 AM, Jun Rao wrote: > > > > Hi, everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer Onur > > Karaman. > > > > Onur's most

Stream Processing Meetup@LinkedIn (Dec 4th)

Hi Kafka users and developers, We are going to host our quarterly Stream Processing Meetup@LinkedIn on Dec 4. There will be three speakers from Slack, Uber and LinkedIn. Please check the details below if you are interested. Thanks, Jiangjie (Becket) Qin *Stream Processing with Apache Kafka

Re: Stream Processing Meetup@LinkedIn (Dec 4th)

Hi Paolo, Yes, we will stream the meetup. Usually the link will be posted to the meetup website a couple of hours before the meetup. Feel free to ping me if you don't see it. Thanks, Jiangjie (Becket) Qin On Fri, Nov 17, 2017 at 11:59 AM, Paolo Patierno wrote: > Hi Becket, > I w

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations, Rajini! On Wed, Jan 17, 2018 at 1:52 PM, Ismael Juma wrote: > Congratulations Rajini! > > On 17 Jan 2018 10:49 am, "Gwen Shapira" wrote: > > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since then, she remained > active in the com

Streams meetup@LinkedIn (Mar 21)

check the details below if you are interested. https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/ <https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/> Thanks, Jiangjie (Becket) Qin

Re: Streams meetup@LinkedIn (Mar 21)

Software Engineer, LinkedIn) 7:55-8:30 PM: Building Venice with Apache Kafka & Samza (Gaojie Liu, Senior Software Engineer, LinkedIn) Please use the following link to RSVP if you are interested. https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/ Thanks, Jiangjie (Becket)

[ANNOUNCE] New Committer: Dong Lin

discussions and doing code reviews. Congratulations and looking forward to your future contribution, Dong! Jiangjie (Becket) Qin, on behalf of Apache Kafka PMC

Re: Messages corrupted in kafka

some other clients? Jiangjie (Becket) Qin On Wed, Mar 23, 2016 at 8:28 PM, sunil kalva wrote: > can some one help me out here. > > On Wed, Mar 23, 2016 at 7:36 PM, sunil kalva > wrote: > > > Hi > > I am seeing few messages getting corrupted in kafka, It is not h

Re: Messages corrupted in kafka

er i cant read and replay them again. > > On Thu, Mar 24, 2016 at 9:42 AM, Becket Qin wrote: > > > Hi Sunil, > > > > The messages in Kafka has a CRC stored with each of them. When consumer > > receives a message, it will compute the CRC from the message bytes an

Re: [ANNOUCE] Apache Kafka 0.10.0.0 Released

Awesome! On Tue, May 24, 2016 at 9:41 AM, Jay Kreps wrote: > Woohoo!!! :-) > > -Jay > > On Tue, May 24, 2016 at 9:24 AM, Gwen Shapira wrote: > > > The Apache Kafka community is pleased to announce the release for Apache > > Kafka 0.10.0.0. > > This is a major release with exciting new features,

Re: KIP-33 Opt out from Time Based indexing

that the previous behavior is maintained, because for users it would be bad if upgrading cause their message got deleted due the change of the behavior. What do you think? Thanks, Jiangjie (Becket) Qin On Thu, Aug 25, 2016 at 2:36 PM, Jun Rao wrote: > Jan, > > Thanks a lot for the

Re: KIP-33 Opt out from Time Based indexing

means "Max timestamp up util this offset", which is also aligned with offset index entry. Thanks, Jiangjie (Becket) Qin On Fri, Aug 26, 2016 at 10:29 AM, Jun Rao wrote: > Jiangjie, > > I am not sure about changing the default to LogAppendTime since CreateTime > is probably wha

Re: KIP-33 Opt out from Time Based indexing

al position directly :) Thanks, Jiangjie (Becket) Qin On Fri, Aug 26, 2016 at 3:41 PM, Jan Filipiak wrote: > Hi Jun, > > thanks for taking the time to answer on such a detailed level. You are > right Log.fetchOffsetByTimestamp works, the comment is just confusing > "// Get all

Re: KIP-33 Opt out from Time Based indexing

Hi Jun, I just created KAFKA-4099 and will submit patch soon. Thanks, Jiangjie (Becket) Qin On Mon, Aug 29, 2016 at 11:55 AM, Jun Rao wrote: > Jiangjie, > > Good point on the time index format related to uncompressed messages. It > does seem that indexing based on file positio

Re: [ANNOUNCE] New committer: Jason Gustafson

Congrats, Jason! On Tue, Sep 6, 2016 at 5:09 PM, Onur Karaman wrote: > congrats jason! > > On Tue, Sep 6, 2016 at 4:12 PM, Sriram Subramanian > wrote: > > > Congratulations Jason! > > > > On Tue, Sep 6, 2016 at 3:40 PM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com > > > wrote: > > > > > Co

Re: Does Kafka 0.9 can guarantee not loss data

In order to satisfy a produce response, there are two conditions: A. The leader's high watermark should be higher than the requiredOffset (max offset in that produce request of that partition) B. The number of in sync replica is greater than min.isr. The ultimate goal here is to make sure at least

Re: Snazzy new look to our website

Much fancier now :) On Tue, Oct 4, 2016 at 5:51 PM, Ali Akhtar wrote: > Just noticed this on pulling up the documentation. Oh yeah! This new look > is fantastic. > > On Wed, Oct 5, 2016 at 4:31 AM, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > wrote: > > > +1 > > > > Thank you for the much

Re: client use high cpu which caused by delayedFetch operation immediately return

Can you check if you have KAFKA-3003 when you run the code? On Sat, Oct 8, 2016 at 12:52 AM, Kafka wrote: > Hi all, > we found our consumer have high cpu load in our product > enviroment，as we know，fetch.min.bytes and fetch.wait.ma < > http://fetch.wait.ma/>x.ms will affect the frequency

Re: client use high cpu which caused by delayedFetch operation immediately return

Glad to know :) On Tue, Oct 18, 2016 at 1:24 AM, Json Tu wrote: > Thanks. I patch it, and everything goes ok. > > 在 2016年10月9日，下午12:39，Becket Qin 写道： > > > > Can you check if you have KAFKA-3003 when you run the code? > > > > On Sat, Oct 8, 2016 at 12:52

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a > > committer and we are pleased to announce that he has accepted! > > > > Becket has made significant contributions to Kafka over the last two > years. > > He has been deeply involved in a broad range of KI

Re: log compaction

consumption may be higher. Given you are still running on 0.8, there are a few additional things to be aware of: 3. Log compaction doesn't work with compressed topics until Kafka 0.9.0. 4. Some other potential issues that may caused by log compaction. (e.g. KAFKA-2024). Jiangjie (Becket) Qin

Re: Mysterious timeout

hat case. Could you check if the cause of the issue you saw matches one of the above? Thanks, Jiangjie (Becket) Qin On Fri, Nov 4, 2016 at 11:52 AM, Jeff Widman wrote: > Mike, > Did you ever figure this out? > > We're considering using Kafka on Kubernetes and very intereste

Re: Re-consume messages

moved. In the worst case, you may consume a lot more messages than you want to, but you should not miss any messages. Jiangjie (Becket) Qin On Sun, Nov 6, 2016 at 1:30 AM, Amit K wrote: > Hi All, > > I am using kafka 0.9.0.1 with high level java producer and consumer. > I need to h

Re: Checking the consumer lag when using manual partition assignment with the KafkaConsumer

Hi Robert, Have you tried Kafka-consumer-offset-checker.sh come with Kafka? It should work regardless whether Kafka based group management is used or not. Thanks, Jiangjie (Becket) Qin On Sun, Nov 6, 2016 at 4:00 PM, Vincent Dautremont < vincent.dautrem...@olamobile.com> wrote: > By

Re: Checking the consumer lag when using manual partition assignment with the KafkaConsumer

It query broker first to see if there is committed offset for the partition, if there isn't, it then fall back to looking at ZK. Thanks, Jiangjie (Becket) Qin On Sun, Nov 6, 2016 at 6:36 PM, Matthias J. Sax wrote: > kafka-consumer-offet-checker works for older Kakfa versions for which >

Re: Deadlock using latest 0.10.1 Kafka release

ucing with acks=all. So if you are blocking on this issue maybe you can pick up the patch as a short term solution. Meanwhile we will prioritize the ticket. Thanks, Jiangjie (Becket) Qin On Mon, Nov 7, 2016 at 9:47 AM, Marcos Juarez wrote: > We ran into this issue several more times over the

Re: [ANNOUNCE] New Committer: Randall Hauch

Congratulations, Randall! On Sat, Feb 16, 2019 at 2:44 AM Matthias J. Sax wrote: > Congrats Randall! > > > -Matthias > > On 2/14/19 6:16 PM, Guozhang Wang wrote: > > Hello all, > > > > The PMC of Apache Kafka is happy to announce another new committer > joining > > the project today: we have inv

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Congrats, Matthias! On Sat, Apr 20, 2019 at 10:28 AM Matthias J. Sax wrote: > Thank you all! > > -Matthias > > > On 4/19/19 3:58 PM, Lei Chen wrote: > > Congratulations Matthias! Well deserved! > > > > -Lei > > > > On Fri, Apr 19, 2019 at 2:55 PM James Cheng > > wro

Re: Deadlock using latest 0.10.1 Kafka release

0.10.0.1 code There might be a bunch of .timeindex file left over but that should be fine. Thanks, Jiangjie (Becket) Qin On Fri, Nov 11, 2016 at 9:51 AM, Marcos Juarez wrote: > Becket/Jason, > > We deployed a jar with the base 0.10.1.0 release plus the KAFKA-3994 patch, > but we&#

Re: [ANNOUNCE] New committer: Grant Henke

Congrats Grant! On Wed, Jan 11, 2017 at 2:17 PM, Kaufman Ng wrote: > Congrats Grant! > > On Wed, Jan 11, 2017 at 4:28 PM, Jay Kreps wrote: > > > Congrats Grant! > > > > -Jay > > > > On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira > wrote: > > > > > The PMC for Apache Kafka has invited Grant Hen

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

Thanks Ewen :) On Wed, Feb 22, 2017 at 5:15 AM, Kenny Gorman wrote: > We are excited about this release! Excellent work! > > Thanks > Kenny Gorman > www.eventador.io > > > On Feb 22, 2017, at 2:33 AM, Ewen Cheslack-Postava > wrote: > > > > The Apache Kafka community is pleased to announce the r

Re: [ANNOUNCE] New committer: Rajini Sivaram

Congratulations! Rajini! Great work! On Mon, Apr 24, 2017 at 3:33 PM, Jason Gustafson wrote: > Woohoo! Great work, Rajini! > > On Mon, Apr 24, 2017 at 3:27 PM, Jun Rao wrote: > > > Congratulations, Rajini ! Thanks for all your contributions. > > > > Jun > > > > On Mon, Apr 24, 2017 at 2:06 PM,

Streams Processing Meetup @LinkedIn

Hello! LinkedIn will be hosting a Streams Processing meetup on Wednesday May 24, 6-9PM in our Sunnyvale HQ. We'll have 3 exciting talks planned: - Streaming Data Pipelines with Brooklin - Kafka at Half the Price