Re: [kafka-clients] Re: [VOTE] 3.2.0 RC0

2022-04-26 Thread Jun Rao
ll abort the voting and create a new release candidate. > > Best, > Bruno > > On 26.04.22 18:09, 'Jun Rao' via kafka-clients wrote: > > Hi, Bruno, > > > > Could we include https://github.com/apache/kafka/pull/12064 > > <https://github.com/apache/

Re: [VOTE] 3.2.0 RC0

2022-04-26 Thread Jun Rao
Hi, Bruno, Could we include https://github.com/apache/kafka/pull/12064 in 3.2.0? This fixes an issue introduced in 3.2.0 where in some of the error cases, the producer interceptor is called twice for the same record. Thanks, Jun On Tue, Apr 26, 2022 at 6:34 AM Bruno Cadonna wrote: > Hi all, >

Re: CVE-2021-44228 – Log4j 2 Vulnerability

2021-12-14 Thread Jun Rao
Hi, Everyone, Just to provide an update. https://kafka.apache.org/cve-list is now updated with this CVE. Thanks, Jun On Tue, Dec 14, 2021 at 3:30 PM Jun Rao wrote: > Hi, Israel, > > Randall added some clarification for the connectors in the PR. > > Thanks, > > Jun > &

Re: CVE-2021-44228 – Log4j 2 Vulnerability

2021-12-14 Thread Jun Rao
not use this dependency, it is possible > external connectors that use it could introduce vulnerabilities if they > depend on the affected log4j2 version > > > On Tue, Dec 14, 2021 at 1:40 PM Israel Ekpo wrote: > > > Sure I will take a look at it shortly > > > > O

Re: CVE-2021-44228 – Log4j 2 Vulnerability

2021-12-14 Thread Jun Rao
Hi, Luke, Thanks for the analysis. We are trying to put a public statement on this through this PR: https://github.com/apache/kafka-site/pull/388. If anyone has more feedback, we can iterate on the PR. Thanks, Jun On Tue, Dec 14, 2021 at 7:53 AM Murilo Tavares wrote: > What about Kafka-Conne

Re: updates on Kafka Summit in 2020

2020-04-28 Thread Jun Rao
5, 2020 at 4:52 PM Jun Rao wrote: > Hi, Everyone, > > Here is an update on the upcoming Kafka Summit events in 2020. > > 1. Unfortunately, Kafka Summit London, originally planned on Apr 27/28, > has been cancelled due to COVID-19. > > 2. Kafka Summit Austin (Aug 24/

updates on Kafka Summit in 2020

2020-04-15 Thread Jun Rao
Hi, Everyone, Here is an update on the upcoming Kafka Summit events in 2020. 1. Unfortunately, Kafka Summit London, originally planned on Apr 27/28, has been cancelled due to COVID-19. 2. Kafka Summit Austin (Aug 24/25) is still on. The CFP ( https://events.kafka-summit.org/kafka-summit-austin-2

Re: [kafka-clients] [VOTE] 2.5.0 RC3

2020-04-13 Thread Jun Rao
Hi, David, Thanks for running the release. Verified quickstart on the scala 2.12 binary. +1 from me. Jun On Tue, Apr 7, 2020 at 9:03 PM David Arthur wrote: > Hello Kafka users, developers and client-developers, > > This is the forth candidate for release of Apache Kafka 2.5.0. > > * TLS 1.3 su

[ANNOUNCE] New committer: Mickael Maison

2019-11-07 Thread Jun Rao
Hi, Everyone, The PMC of Apache Kafka is pleased to announce a new Kafka committer Mickael Maison. Mickael has been contributing to Kafka since 2016. He proposed and implemented multiple KIPs. He has also been propomating Kafka through blogs and public talks. Congratulations, Mickael! Thanks,

Re: Add me to the contributors list...

2019-10-10 Thread Jun Rao
Hi, Senthil, Thanks for your interest. Just added you to the contributors list and gave you the wiki permissions. Jun On Thu, Oct 10, 2019 at 5:26 PM Senthilnathan Muthusamy wrote: > Hi, > > I am Senthil from Microsoft Azure Compute and will be contributing to the > KIP-280. Can you please add

[ANNOUNCE] New Kafka PMC member: Sriharsh Chintalapan

2019-04-18 Thread Jun Rao
Hi, Everyone, Sriharsh Chintalapan has been active in the Kafka community since he became a Kafka committer in 2015. I am glad to announce that Harsh is now a member of Kafka PMC. Congratulations, Harsh! Jun

Re: [kafka-clients] [VOTE] 2.1.1 RC2

2019-02-14 Thread Jun Rao
Hi, Colin, Thanks for running the release. Verified the quickstart for 2.12 binary. +1 from me. Jun On Fri, Feb 8, 2019 at 12:02 PM Colin McCabe wrote: > Hi all, > > This is the third candidate for release of Apache Kafka 2.1.1. This > release includes many bug fixes for Apache Kafka 2.1. > >

Hackathons with funding of the EU

2019-02-12 Thread Jun Rao
Hi Guys, I am passing along the following info from EU. If you are interested in the event, please contact the coordinator. Thanks. The EU via its EU-FOSSA 2 project have invited a number of communities including Apache Kafka to consider taking one of their 3 planned Hackathons this year, to be h

Re: [ANNOUNCE] New Committer: Vahid Hashemian

2019-01-15 Thread Jun Rao
Congratulations, Vahid. Thanks, Jun On Tue, Jan 15, 2019 at 2:45 PM Jason Gustafson wrote: > Hi All, > > The PMC for Apache Kafka has invited Vahid Hashemian as a project > committer and > we are > pleased to announce that he has accepted! > > Vahid has made numerous contributions to the Kafka

Re: Kafka Summit NYC and London in 2019

2018-12-17 Thread Jun Rao
Hi, Everyone, This is a reminder about the deadline for proposal this Thursday. Thanks, Jun On Tue, Dec 4, 2018 at 1:49 PM Jun Rao wrote: > Hi, Everyone, > > We have two upcoming Kafka Summits, one in NYC and another in London. The > deadline for summiting proposals is Dec 20 for

Kafka Summit NYC and London in 2019

2018-12-04 Thread Jun Rao
Hi, Everyone, We have two upcoming Kafka Summits, one in NYC and another in London. The deadline for summiting proposals is Dec 20 for both events. Please consider submitting a proposal if you are interested. The Links to submit abstracts are Kafka Summit NYC - https://myeventi.events/kafka19/ny/

Re: [ANNOUNCE] Apache Kafka 2.0.1

2018-11-09 Thread Jun Rao
Thanks for running the release, Mani! Jun On Fri, Nov 9, 2018 at 11:42 AM, Manikumar wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 2.0.1 > > This is a bug fix release and it includes fixes and improvements from 51 > JIRAs. > > All of the changes in th

Re: [kafka-clients] [VOTE] 2.0.1 RC0

2018-11-05 Thread Jun Rao
Hi, Mani, Thanks for running the release. Verified quickstart on 2.12 binary. +1 Jun On Thu, Oct 25, 2018 at 7:28 PM, Manikumar wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 2.0.1. > > This is a bug fix release closing

Re: [ANNOUNCE] New Committer: Manikumar Reddy

2018-10-11 Thread Jun Rao
Congratulations, Mani! Jun On Thu, Oct 11, 2018 at 10:39 AM, Jason Gustafson wrote: > Hi all, > > The PMC for Apache Kafka has invited Manikumar Reddy as a committer and we > are > pleased to announce that he has accepted! > > Manikumar has contributed 134 commits including significant work to

Re: [kafka-clients] [VOTE] 1.0.2 RC1

2018-07-02 Thread Jun Rao
Hi, Matthias, Thanks for the running the release. Verified quickstart on scala 2.12 binary. +1 Jun On Fri, Jun 29, 2018 at 10:02 PM, Matthias J. Sax wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for release of Apache Kafka 1.0.2. > > This is a b

Re: [kafka-clients] [VOTE] 1.1.1 RC2

2018-06-29 Thread Jun Rao
Hi, Dong, Thanks for running the release. Verified quickstart on scala 2.12 binary. +1 Jun On Thu, Jun 28, 2018 at 6:12 PM, Dong Lin wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for release of Apache Kafka 1.1.1. > > Apache Kafka 1.1.1 is a bug

Re: [kafka-clients] [VOTE] 0.11.0.3 RC0

2018-06-29 Thread Jun Rao
Hi, Matthias, Thanks for running the release. Verified quickstart on scala 2.12 binary. +1 Jun On Fri, Jun 22, 2018 at 3:14 PM, Matthias J. Sax wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 0.11.0.3. > > This is a bug

Re: [kafka-clients] [VOTE] 0.10.2.2 RC1

2018-06-29 Thread Jun Rao
Hi, Matthias, Thanks for running the release. Verified quickstart on scala 2.12 binary. +1 Jun On Fri, Jun 22, 2018 at 6:43 PM, Matthias J. Sax wrote: > Hello Kafka users, developers and client-developers, > > This is the second candidate for release of Apache Kafka 0.10.2.2. > > Note, that RC

CFP: Kafka Summit San Francisco, 2018

2018-05-08 Thread Jun Rao
Hi, Everyone, Just want to let you know that Kafka Summit San Francisco, 2018 is open for submissions. The deadline for submission is Jun. 16. The conference itself will be held on Oct. 16 - Oct. 17. More details can be found at https://kafka-summit.org/kafka-summit-san-francisco-2018/speakers/

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC4

2018-03-27 Thread Jun Rao
Hi, Rajini, Thanks for running the release. Verified quickstart on scala 2.11 binary. +1 Jun On Tue, Mar 27, 2018 at 6:22 AM, Rajini Sivaram wrote: > Can we get some more votes for this RC so that the release can be rolled > out soon? > > Many thanks, > > Rajini > > On Sat, Mar 24, 2018 at 6:

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC0

2018-03-01 Thread Jun Rao
KAFKA-6111 is now merged to 1.1 branch. Thanks, Jun On Thu, Mar 1, 2018 at 2:50 PM, Jun Rao wrote: > Hi, Damian, > > It would also be useful to include KAFKA-6111, which prevents > deleteLogDirEventNotifications path to be deleted correctly from > Zookeeper. The patch shou

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC0

2018-03-01 Thread Jun Rao
Hi, Damian, It would also be useful to include KAFKA-6111, which prevents deleteLogDirEventNotifications path to be deleted correctly from Zookeeper. The patch should be committed later today. Thanks, Jun On Thu, Mar 1, 2018 at 1:47 PM, Damian Guy wrote: > Thanks Jason. Assuming the system te

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

2018-01-18 Thread Jun Rao
Congratulations, Rajini ! Jun On Wed, Jan 17, 2018 at 10:48 AM, Gwen Shapira wrote: > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since then, she remained > active in the community and contributed major patches, reviews and KIP > discussions. I a

Re: Kafka Replicated Partition Limits

2018-01-03 Thread Jun Rao
I only tracked outstanding requests thus far from ZK-side of > things. At 9.5k topics, I recorded about 5k outstanding requests. I'll > start tracking this better for my next run. Anything else worth > tracking? > > Jun Rao: > I'm testing the latest 1.0.0. I'm testing nor

Re: Kafka Replicated Partition Limits

2018-01-03 Thread Jun Rao
Hi, Andrey, Thanks for reporting the results. Which version of Kafka are you testing? Also, it would be useful to know if you are testing the normal mode when all replicas are up and in sync, or the failure mode when some of the replicas are being restarted. Typically, ZK is only accessed in the f

Re: [ANNOUNCE] New committer: Onur Karaman

2017-11-07 Thread Jun Rao
ted; I am > assuming these bugs are still valid for the current stable release. > > Affan > > - Affan > > On Mon, Nov 6, 2017 at 10:24 PM, Jun Rao wrote: > > > Hi, everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer Onur >

[ANNOUNCE] New committer: Onur Karaman

2017-11-06 Thread Jun Rao
Hi, everyone, The PMC of Apache Kafka is pleased to announce a new Kafka committer Onur Karaman. Onur's most significant work is the improvement of Kafka controller, which is the brain of a Kafka cluster. Over time, we have accumulated quite a few correctness and performance issues in the control

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 1.0.0 Released

2017-11-01 Thread Jun Rao
Guozhang, Thanks for running the release! Jun On Wed, Nov 1, 2017 at 7:27 AM, Guozhang Wang wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 1.0.0. > > This is a major release of the Kafka project, and is no mere bump of the > version number. The Apache

Re: [kafka-clients] [VOTE] 1.0.0 RC2

2017-10-18 Thread Jun Rao
Hi, Guozhang, Thanks for running the release. Just a quick clarification. The statement that "* Controller improvements: async ZK access for faster administrative request handling" is not accurate. What's included in 1.0.0 is a logging change improvement in the controller, which does give signific

Re: [kafka-clients] Re: [VOTE] 1.0.0 RC1

2017-10-16 Thread Jun Rao
Hi, Guozhang, Onur found an existing performance bug in the controller when there are lots of partitions. The fix is simple ( https://github.com/apache/kafka/pull/4075) and reduces the controlled shutdown time from 6.5 mins to 30 secs, with 25K partitions, RF=2 and 5 brokers. It would be useful t

Re: [VOTE] 0.11.0.1 RC0

2017-09-11 Thread Jun Rao
Hi, Damian, Thanks for running the release. Verified the quickstart from the src distribution. +1 Jun On Tue, Sep 5, 2017 at 1:34 PM, Damian Guy wrote: > Hello Kafka users, developers and client-developers, > > This is the first candidate for release of Apache Kafka 0.11.0.1. > > This is a bug

Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

2017-06-28 Thread Jun Rao
Powers, Davor Poldrugo, dejan2609, > Dhwani Katagade, Dong Lin, Dustin Cote, Edoardo Comar, Eno Thereska, Ewen > Cheslack-Postava, gosubpl, Grant Henke, Guozhang Wang, Gwen Shapira, > Hamidreza Afzali, Hao Chen, hejiefang, Hojjat Jafarpour, huxi, Ismael Juma, > Ivan A. Melnikov, Jai

Re: [kafka-clients] Re: [VOTE] 0.11.0.0 RC2

2017-06-26 Thread Jun Rao
Hi, Ismael, Thanks for running the release. +1. Verified quickstart on the 2.11 binary. Jun On Mon, Jun 26, 2017 at 3:53 PM, Ismael Juma wrote: > Hi Vahid, > > There are a few known issues when running Kafka on Windows. A PR with some > fixes is: https://github.com/apache/kafka/pull/3283. The

Re: Data loss after a Kafka Broker restart scenario

2017-05-24 Thread Jun Rao
Hi, Fathima, Yes, the most efficient way to verify if a message is sent successfully is through the producer callback. You can take a look at PrintInfoCallback in org.apache.kafka.toolsVerifiableProducer as an example. Our system tests use that to verify if any data loss has occurred. Thanks, Ju

Re: Data loss after a Kafka Broker restart scenario

2017-05-19 Thread Jun Rao
Hi, Fathima, Did you check if produced messages are acked successfully? Only successfully acked messages are guaranteed to be preserved on the broker. Thanks, Jun On Thu, May 18, 2017 at 12:10 AM, Fathima Amara wrote: > Hi Jun, > > Thanks alot for the reply. As suggested, I tried running my a

Re: Data loss after a Kafka Broker restart scenario

2017-05-16 Thread Jun Rao
Hi, Fathima, There is a known data loss issue that's described in KIP-101 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-101+-+Alter+Replication+Protocol+to+use+Leader+Epoch+rather+than+High+Watermark+for+Truncation). The issue happens rarely, but has been exposed in some of our system te

Re: [VOTE] 0.10.2.1 RC3

2017-04-26 Thread Jun Rao
Hi, Gwen, Thanks for doing the release. +1 from me. Jun On Fri, Apr 21, 2017 at 9:56 AM, Gwen Shapira wrote: > Hello Kafka users, developers, friends, romans, countrypersons, > > This is the fourth (!) candidate for release of Apache Kafka 0.10.2.1. > > It is a bug fix release, so we have lots

Re: [ANNOUNCE] New committer: Rajini Sivaram

2017-04-24 Thread Jun Rao
Congratulations, Rajini ! Thanks for all your contributions. Jun On Mon, Apr 24, 2017 at 2:06 PM, Gwen Shapira wrote: > The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we > are pleased to announce that she has accepted! > > Rajini contributed 83 patches, 8 KIPs (all secur

Re: Re: Re: ZK and Kafka failover testing

2017-04-20 Thread Jun Rao
K had a long GC pause and Kafka > > lost connection temporarily. The brokers kept sending data just fine > > for existing topics. However, when ZK came back, the kafka cluster > > could not recover gracefully because of this issue: > > https://issues.apache.org/ > > jira/browse

Re: Re: ZK and Kafka failover testing

2017-04-19 Thread Jun Rao
Hi, Shri, As Onur explained, if ZK is down, Kafka can still work, but won't be able to react to actual broker failures until ZK is up again. So if a broker is down in that window, some of the partitions may not be ready for read or write. As for the duplicates in the consumer, Hans had a good poi

new mail archive service from Apache

2017-03-01 Thread Jun Rao
Hi, Just want to pass along this to the community. There is a new mail archive service https://lists.apache.org. It's beta, but is the long-term solution for official ASF mail archives, and offers much better searching/threading than mail-archives.a.o does. Thanks, Jun

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

2017-02-22 Thread Jun Rao
ard Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen > Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang, > Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour, > huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason > Gust

Re: [kafka-clients] [VOTE] 0.10.2.0 RC2

2017-02-15 Thread Jun Rao
Hi, Ewen, Thanks for running the release. +1. Verified quickstart on 2.10 binary. Jun On Tue, Feb 14, 2017 at 10:39 AM, Ewen Cheslack-Postava wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for release of Apache Kafka 0.10.2.0. > > This is a minor

Re: Quick replication question - loss of committed messages during preferred replica election

2017-01-24 Thread Jun Rao
ze, false, Some(e)) > > The consumer then restarts back to the beginning. This looks to be the > source of our 'data loss', which isn't actually loss but a bad > interaction of failover and catching a stale HWM leading to errors being > thrown by the broker when

Re: timeindex file timestamp mismatch (0.10.1.0)

2017-01-20 Thread Jun Rao
Meghana, You are probably seeing this when running the DumpLogSegment tool on the active (last) log segment. DumpLogSegment tool is supposed to only be used on the index of the immutable segments (i.e., when they are rolled). We preallocate the index on the active segment with 0 values. So, DumpLo

Re: Quick replication question - loss of committed messages during preferred replica election

2017-01-18 Thread Jun Rao
Hi, Mark, Your understanding about HWM transition is correct. When the leader changes due to preferred leader election, the new leader will have all the committed messages, but a potentially stale HWM. The new leader won't do any truncation to its local log though. Instead, it will try to commit a

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Jun Rao
Grant, Thanks for all your contribution! Congratulations! Jun On Wed, Jan 11, 2017 at 2:51 PM, Gwen Shapira wrote: > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews,

Re: [VOTE] Vote for KIP-101 - Leader Epochs

2017-01-05 Thread Jun Rao
. It is really a separate issue with wider implications. > I'd be happy to add KAFKA-1120 into the release though if we have time. > 4) Agreed. Not sure exactly how that's going to play out, but I think we're > on the same page. > > Please could > > Cheers > B

Re: [VOTE] Vote for KIP-101 - Leader Epochs

2017-01-04 Thread Jun Rao
Hi, Ben, Thanks for the proposal. Looks good overall. A few comments below. 1. For LeaderEpochRequest, we need to include topic right? We probably want to follow other requests by nesting partition inside topic? For LeaderEpochResponse, do we need to return leader_epoch? I was thinking that we co

Re: [VOTE] 0.10.1.1 RC1

2016-12-19 Thread Jun Rao
Hi, Guozhang, Thanks for preparing the release. Verified quickstart. +1 Jun On Thu, Dec 15, 2016 at 1:29 PM, Guozhang Wang wrote: > Hello Kafka users, developers and client-developers, > > This is the second, and hopefully the last candidate for the release of > Apache Kafka 0.10.1.1 before th

Re: Investigating apparent data loss during preferred replica election

2016-11-21 Thread Jun Rao
sion about > multiple-failovers is germane to the problem we saw. Each of our partitions > only had a single failover, and yet 4 of them still truncated committed > data. > > -- > Mark Smith > m...@qq.is > > > On Mon, Nov 21, 2016, at 05:12 PM, Jun Rao wrote: > > Hi

Re: Investigating apparent data loss during preferred replica election

2016-11-21 Thread Jun Rao
does, though. I don't see a step in the ticket > about become-leader making a call to the old leader to get the latest > generation snapshot? > > -- > Mark Smith > m...@qq.is > > On Fri, Nov 18, 2016, at 10:52 AM, Jun Rao wrote: > > Mark, > > > > Thanks for

Re: Investigating apparent data loss during preferred replica election

2016-11-18 Thread Jun Rao
Mark, Thanks for reporting this. First, a clarification. The HW is actually never advanced until all in-sync followers have fetched the corresponding message. For example, in step 2, if all follower replicas issue a fetch request at offset 10, it serves as an indication that all replicas have rece

Re: Segments being deleted too early after upgrading 0.9.0.1 to 0.10.1.0

2016-11-01 Thread Jun Rao
the past... > > On Mon, Oct 31, 2016 at 5:02 PM, Jun Rao wrote: > > > Hi, James, > > > > Thanks for testing and reporting this. What you observed is actually not > > the expected behavior in 0.10.1 based on the design. The way that > retention > >

Re: Segments being deleted too early after upgrading 0.9.0.1 to 0.10.1.0

2016-10-31 Thread Jun Rao
Hi, James, Thanks for testing and reporting this. What you observed is actually not the expected behavior in 0.10.1 based on the design. The way that retention works in 0.10.1 is that if a log segment has at least one message with a timestamp, we will use the largest timestamp in that segment to d

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

2016-10-31 Thread Jun Rao
Congratulations, Jiangjie. Thanks for all your contributions to Kafka. Jun On Mon, Oct 31, 2016 at 10:35 AM, Joel Koshy wrote: > The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a > committer and we are pleased to announce that he has accepted! > > Becket has made significa

Re: [VOTE] Add REST Server to Apache Kafka

2016-10-25 Thread Jun Rao
Sorry a typo. -1 instead. Thanks, Jun On Tue, Oct 25, 2016 at 4:24 PM, Jun Rao wrote: > +1 > > Thanks, > > Jun > > On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapani > wrote: > >> Hi All, >>We are proposing to have a REST Server as part

Re: [VOTE] Add REST Server to Apache Kafka

2016-10-25 Thread Jun Rao
+1 Thanks, Jun On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapani wrote: > Hi All, >We are proposing to have a REST Server as part of Apache Kafka > to provide producer/consumer/admin APIs. We Strongly believe having > REST server functionality with Apache Kafka will help a lot

Re: [kafka-clients] [VOTE] 0.10.1.0 RC3

2016-10-17 Thread Jun Rao
Thanks for preparing the release. Verified quick start on scala 2.11 binary. +1 Jun On Fri, Oct 14, 2016 at 4:29 PM, Jason Gustafson wrote: > Hello Kafka users, developers and client-developers, > > One more RC for 0.10.1.0. We're hoping this is the final one so that we > can meet the release t

Re: KIP-33 Opt out from Time Based indexing

2016-09-06 Thread Jun Rao
t need to bother > that much. > > Best Jan > > > > > > On 29.08.2016 19:36, Jun Rao wrote: > >> Jan, >> >> For the usefulness of time index, it's ok if you don't plan to use it. >> However, I do think there are other people who will wan

Re: Kafka 0.10.0 MetricsReporter implementation only reporting zeros

2016-09-06 Thread Jun Rao
Michael, One thing to be aware is that if the producer stops sending messages, some of the metrics will be reset to 0 after the metric window elapses. Do you see metric values reported in the reporter different from that in jconsole? Thanks, Jun On Tue, Sep 6, 2016 at 3:03 PM, Michael Ross wro

Re: KIP-33 Opt out from Time Based indexing

2016-08-29 Thread Jun Rao
offset", which is also aligned with offset index entry. > > Thanks, > > Jiangjie (Becket) Qin > > On Fri, Aug 26, 2016 at 10:29 AM, Jun Rao wrote: > > > Jiangjie, > > > > I am not sure about changing the default to LogAppendTime since > CreateTime > &

Re: KIP-33 Opt out from Time Based indexing

2016-08-29 Thread Jun Rao
ew > hundreds seeks for bigger topics. but I guess the broker does these extra > for the new index file now. > > This index, I feel is just not following the whole "kafka-way". Can you > suggest on the proposed re-factoring? what are the chance to get it > upstream if I co

Re: KIP-33 Opt out from Time Based indexing

2016-08-26 Thread Jun Rao
.timestamp.type. I am wondering would it be less surprising if > we change the default value to LogAppendTime so that the previous behavior > is maintained, because for users it would be bad if upgrading cause their > message got deleted due the change of the behavior. What do you thi

Re: Networking errors and durability settings

2016-08-26 Thread Jun Rao
Bryan, Were there multiple brokers losing ZK session around the same time? There is one known issue https://issues.apache.org/jira/browse/KAFKA-1211. Basically, if the leader changes too quickly, it's possible for a follower to truncate some previous committed messages and then immediately becomes

Re: Networking errors and durability settings

2016-08-25 Thread Jun Rao
Bryan, https://issues.apache.org/jira/browse/KAFKA-3410 reported a similar issue but only happened when the leader broker's log was manually deleted. In your case, was there any data loss in the broker due to things like power outage? Thanks, Jun On Tue, Aug 23, 2016 at 9:00 AM, Bryan Baugher

Re: KIP-33 Opt out from Time Based indexing

2016-08-25 Thread Jun Rao
to have this > log.message.timestamp.difference.max.ms take effect. I think this defeats > goal 1 again. > > ITE having this index in the brokers now feels wired to me. Gives me a > feeling of complexity that I don't need and have a hard time figuring out > how much other peopl

Re: Re: KIP-33 Opt out from Time Based indexing

2016-08-24 Thread Jun Rao
Jan, Thanks for the reply. I actually wasn't sure what your main concern on time-based rolling is. Just a couple of clarifications. (1) Time-based rolling doesn't control how long a segment will be retained for. For retention, if you use time-based, it will now be based on the timestamp in the mes

Re: KIP-33 Opt out from Time Based indexing

2016-08-22 Thread Jun Rao
Jan, Currently, there is no switch to disable the time based index. There are quite a few use cases of time based index. 1. From KIP-33's wiki, it allows us to do time-based retention accurately. Before KIP-33, the time-based retention is based on the last modified time of each log segment. The

Re: kafka broker is dropping the messages after acknowledging librdkafka

2016-08-19 Thread Jun Rao
an we get the fix for this bug ? > > Thanks. > > Regards, > Mazhar Shaikh > > > On Fri, Aug 19, 2016 at 1:24 AM, Jun Rao wrote: > > > Mazhar, > > > > With ack=1, whether you lose messages or not is not deterministic. It > > depends on the time when the br

Re: kafka broker is dropping the messages after acknowledging librdkafka

2016-08-18 Thread Jun Rao
ooks like kafka_2.11-0.9.0.1 has issues(BUG) during replication. > > Thanks, > > Regards, > Mazhar Shaikh. > > > > On Thu, Aug 18, 2016 at 10:30 PM, Jun Rao wrote: > > > Mazhar, > > > > There is probably a mis-understanding. Ack=-1 (or all) doesn't

Re: kafka broker is dropping the messages after acknowledging librdkafka

2016-08-18 Thread Jun Rao
(2, 4), [topic1,5] -> List(2, 4), [topic1,10] -> List(2, 4), > [topic1,7] -> List(2, 4)), 3 -> Map([topic1,33] -> List(3, 5), [topic1,30] > -> List(3, 5), [topic1,24] -> List(3, 5), [topic1,36] -> List(3, 5), > [topic1,38] -> List(3, 5), [topic1,26] -> List(3, 5), [topi

Re: Issue adding server (0.10.0.0)

2016-08-17 Thread Jun Rao
Jarko, Do you have many topic partitions? Currently, if #partitions * fetched_bytes in the response exceeds 2GB, we will get an integer overflow and weird things can happen. We are trying to address this better in KIP-74. If this is the issue, for now, you can try reducing the fetch size or increa

Re: kafka broker is dropping the messages after acknowledging librdkafka

2016-08-17 Thread Jun Rao
the producer queue will be > exhausted is any kafka broker goes down for long time. > > > Thanks. > > Regards, > Mazhar Shaikh. > > > On Wed, Aug 17, 2016 at 8:11 PM, Jun Rao wrote: > > > Are you using acks=1 or acks=all in the producer? Only the latte

Re: kafka broker is dropping the messages after acknowledging librdkafka

2016-08-17 Thread Jun Rao
Are you using acks=1 or acks=all in the producer? Only the latter guarantees acked messages won't be lost after leader failure. Thanks, Jun On Wed, Aug 10, 2016 at 11:41 PM, Mazhar Shaikh wrote: > Hi Kafka Team, > > I'm using kafka (kafka_2.11-0.9.0.1) with librdkafka (0.8.1) API for > produce

Re: [kafka-clients] [ANNOUCE] Apache Kafka 0.10.0.1 Released

2016-08-11 Thread Jun Rao
Ismael, Thanks for running the release. Jun On Wed, Aug 10, 2016 at 5:01 PM, Ismael Juma wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka 0.10.0.1. > This is a bug fix release that fixes 53 issues in 0.10.0.0. > > All of the changes in this release can

Re: [kafka-clients] [VOTE] 0.10.0.1 RC2

2016-08-05 Thread Jun Rao
Thanks for running the release. +1 Jun On Thu, Aug 4, 2016 at 6:54 AM, Ismael Juma wrote: > Hello Kafka users, developers and client-developers, > > This is the third candidate for the release of Apache Kafka 0.10.0.1. This > is a bug fix release and it includes fixes and improvements from 53 J

Re: [VOTE] 0.10.0.0 RC6

2016-05-19 Thread Jun Rao
Thanks for running the release. +1 from me. Verified the quickstart. Jun On Tue, May 17, 2016 at 10:00 PM, Gwen Shapira wrote: > Hello Kafka users, developers and client-developers, > > This is the seventh (!) candidate for release of Apache Kafka > 0.10.0.0. This is a major release that includ

[ANNOUCE] Apache Kafka 0.9.0.1 Released

2016-02-19 Thread Jun Rao
stava, Geoff Anderson, Grant Henke, Guozhang Wang, Gwen Shapira, Ismael Juma, Jaikiran Pai, James Cheng, Jason Gustafson, Jay Kreps, Jesse Anderson, Jiangjie Qin, Jin Xing, Jun Rao, Kim Christensen, Kishore Senji, Luciano Afranllie, Magnus Edenhill, Maksim Logvinenko, Mayuresh Gharat, Michael Blume,

Re: 0.9.0.1 RC1

2016-02-17 Thread Jun Rao
, Christian Posta wrote: > BTW, what's the etiquette for votes (non-binding) for this community? > welcomed? noise? > happy to see the non-binding votes, I'd like to contribute, just don't want > to pollute the vote call. thoughts? > thanks! > > On Tue, Feb

Re: 0.9.0.1 RC1

2016-02-16 Thread Jun Rao
Thanks everyone for voting. The results are: +1 binding = 4 votes (Ewen Cheslack-Postava, Neha Narkhede, Joel Koshy and Jun Rao) +1 non-binding = 3 votes -1 = 0 votes 0 = 0 votes The vote passes. I will release artifacts to maven central, update the dist svn and download site. Will send out an

Re: 0.9.0.1 RC1

2016-02-16 Thread Jun Rao
Since the unit test failures are transient. +1 from myself. Thanks, Jun On Thu, Feb 11, 2016 at 6:55 PM, Jun Rao wrote: > This is the first candidate for release of Apache Kafka 0.9.0.1. This a > bug fix release that fixes 70 issues. > > Release Notes for the 0.9.0.1 rel

0.9.0.1 RC1

2016-02-11 Thread Jun Rao
This is the first candidate for release of Apache Kafka 0.9.0.1. This a bug fix release that fixes 70 issues. Release Notes for the 0.9.0.1 release https://home.apache.org/~junrao/kafka-0.9.0.1-candidate1/RELEASE_NOTES.html *** Please download, test and vote by Tuesday, Feb. 16, 7pm PT Kafka's K

Re: Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

2016-02-08 Thread Jun Rao
It seems that you put in the wrong port in the following statement. Kafka's default port is 9092. 2181 is for Zookeeper. props.put("bootstrap.servers", "localhost:2181"); Thanks, Jun On Mon, Feb 8, 2016 at 4:06 AM, Bhargav Maddikera < bhargav.maddik...@altimetrik.com> wrote: > Hi, > > I try

Kafka 0.9.0.1 plan

2016-02-05 Thread Jun Rao
Hi, Everyone, We have fixed a few critical bugs since 0.9.0.0 was released and are still investigating a few more issues. The current list of issues tracked for 0.9.0.1 can be found below. Among them, only KAFKA-3159 seems to be critical. https://issues.apache.org/jira/issues/?jql=project%20%3D%2

Re: Stuck consumer with new consumer API in 0.9

2016-01-26 Thread Jun Rao
Rajiv, We haven't released 0.9.0.1 yet. To try the fix, you can build a new client jar off the 0.9.0 branch. Thanks, Jun On Mon, Jan 25, 2016 at 12:03 PM, Rajiv Kurian wrote: > Thanks Jason. We are using an affected client I guess. > > Is there a 0.9.0 client available on maven? My search at

Upcoming Kafka Summit this April

2016-01-07 Thread Jun Rao
Hi, Everyone, I wanted to remind you of a couple of Kafka Summit related deadlines coming up. On Monday Jan 11 the Call for Proposals closes at 11:59pm Pacific Time. We encourage you to participate - share your stories and best practices. Submit your proposal at www.kafka-summit.org On Friday Ja

Re: Security in Kafka

2016-01-06 Thread Jun Rao
Mohit, In 0.9, Kafka supports Kerberos. So, you can authenticate with any directory service that supports Kerberos (e.g., Active Directory). Our default authorization is based on individual users, not on groups though. You can find more a bit more info on security in the following links. http://

Re: MD5 checksum on release

2015-12-30 Thread Jun Rao
"", line); split(line, parts, > ":"); print tolower(parts[2]) }'` > if [ "$SOURCE_CHECKSUM" == "$TARGET_CHECKSUM" ] > ... > fi > > Not a huge deal I just think it would make it easier for folks to use the > former rather than GPG. >

Re: MD5 checksum on release

2015-12-30 Thread Jun Rao
Xavier, The md5 checksum is generated by running "gpg --print-md MD5". Is there a command that generates the output that you wanted? Thanks, Jun On Tue, Dec 29, 2015 at 5:13 PM, Xavier Stevens wrote: > The current md5 checksums of the release downloads all seem to be returning > in an atypica

Re: Why does'nt kafka replicate data automatically if a broker goes down

2015-12-28 Thread Jun Rao
Manju, You understanding is correct. Just added the following FAQ. https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Howtoreplaceafailedbroker ? Thanks, Jun On Sun, Dec 27, 2015 at 11:30 PM, Manjunath Shivanna wrote: > Hi, > > Correct me if I am wrong. I believe kafka does not replic

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

2015-12-21 Thread Jun Rao
any ZK issues on the brokers. There were other topics on the > > > very same brokers that didn't seem to be affected. > > > > > > On Thu, Dec 17, 2015 at 5:46 PM, Jun Rao wrote: > > > > > > > Yes, the new java producer is available in 0

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

2015-12-17 Thread Jun Rao
ka. > > On Thu, Dec 17, 2015 at 2:31 PM, Jun Rao wrote: > > > Hmm, anything special with those 3 topics? Also, the broker log shows > that > > the producer uses ack=0, which means the producer shouldn't get errors > like > > leader not found. Could you cl

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

2015-12-17 Thread Jun Rao
On Thu, Dec 17, 2015 at 12:23 PM, Jun Rao wrote: > > > Are you using the new java producer? > > > > Thanks, > > > > Jun > > > > On Thu, Dec 17, 2015 at 9:58 AM, Rajiv Kurian > wrote: > > > > > Hi Jun, > > > A

Re: Fallout from upgrading to kafka 0.9 from 0.8.2.3

2015-12-17 Thread Jun Rao
Are you using the new java producer? Thanks, Jun On Thu, Dec 17, 2015 at 9:58 AM, Rajiv Kurian wrote: > Hi Jun, > Answers inline: > > On Thu, Dec 17, 2015 at 9:41 AM, Jun Rao wrote: > > > Rajiv, > > > > Thanks for reporting this. > > > > 1

  1   2   3   4   5   6   7   8   9   10   >