Re: changelog topic with in-memory state store

2017-01-11 Thread Damian Guy
Hi, There is no way to enable caching on in-memory-store - by definition it is already cached. However the in-memory store will write each update to the changelog (regardless of context.commit), which seems to be the issue you have? When you say large, how large? Have you tested it and observed t

Interpreting value of JMX metric for partition count

2017-01-11 Thread Abhishek Agrawal
Hello Kafka Users, My kafka cluster has two topics: one with 50 partitions, another with 18 partitions. The JMX bean *kafka.server:name=PartitionCount,type=ReplicaManager*, gives value as 35 when I try to probe using JMXTerm $>get -b kafka.server:name=PartitionCount,type=ReplicaManager Value

Re: Interpreting value of JMX metric for partition count

2017-01-11 Thread Sreejith S
Hi Abhishek, Check this if you are looking for a Java implementation. This will give you all Kafka related metrics. https://github.com/srijiths/kafka-connectors/tree/master/kafka-connect-jmx Regards, Srijith On Wed, Jan 11, 2017 at 8:55 AM, Abhishek Agrawal < abhishek.agrawal.1...@gmail.com> wr

RE: kafka-connect log4j

2017-01-11 Thread Berryman, Eric
Silly mistake on my part. Thank you very much for your time! (I'll submit a pull request as mentioned) -Original Message- From: Gwen Shapira [mailto:g...@confluent.io] Sent: Wednesday, January 11, 2017 1:25 AM To: Users Subject: Re: kafka-connect log4j btw. It bugs me a bit that Co

Re: Interpreting value of JMX metric for partition count

2017-01-11 Thread Gwen Shapira
The metric is "partition count per broker". You didn't mention how many brokers and how many replicas you have for each topic, but if you have 1 replica and 2 brokers, then this looks reasonable. Probably 35 partitions on one broker and 33 on the other or something similar. I'd recommend getting at

Re: [VOTE] Vote for KIP-101 - Leader Epochs

2017-01-11 Thread Ben Stopford
So I believe we can mark this as Accepted. I've updated the KIP page. Thanks for the input everyone. On Fri, Jan 6, 2017 at 9:31 AM Ben Stopford wrote: > Thanks Joel. I'll fix up the pics to make them consistent on nomenclature. > > > B > > On Fri, Jan 6, 2017 at 2:39 AM Joel Koshy wrote: > > (

Re: [VOTE] Vote for KIP-101 - Leader Epochs

2017-01-11 Thread Ben Stopford
Sorry - my mistake. Looks like I still need one more binding vote. Is there a committer out there that could add their vote? B On Wed, Jan 11, 2017 at 6:44 PM Ben Stopford wrote: > So I believe we can mark this as Accepted. I've updated the KIP page. > Thanks for the input everyone. > > On Fri,

Is there a performance problem with new broker + old log.message.format.version + new consumer?

2017-01-11 Thread Jeff Widman
We upgraded our Kafka clusters from 0.8.2.1 to 0.10.0.1, but most of our consumers use older libraries that do not support the new message format. So we set the brokers' log.message.format.version to 0.8.2 while we work on upgrading our consumers. In the meantime, I'm worried about a performance p

Re: [VOTE] Vote for KIP-101 - Leader Epochs

2017-01-11 Thread Ben Stopford
OK - my mistake was mistaken! There is consensus. This KIP has been accepted. On Wed, Jan 11, 2017 at 6:48 PM Ben Stopford wrote: > Sorry - my mistake. Looks like I still need one more binding vote. Is > there a committer out there that could add their vote? > > B > > On Wed, Jan 11, 2017 at 6:4

[VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Ben Stopford
Looks like there was a good consensus on the discuss thread for KIP-106 so lets move to a vote. Please chime in if you would like to change the default for unclean.leader.election.enabled from true to false. https://cwiki.apache.org/confluence/display/KAFKA/%5BWIP%5D+KIP-106+-+Change+Default+uncl

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Jason Gustafson
+1 On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote: > Looks like there was a good consensus on the discuss thread for KIP-106 so > lets move to a vote. > > Please chime in if you would like to change the default for > unclean.leader.election.enabled from true to false. > > https://cwiki.apa

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Onur Karaman
+1 On Wed, Jan 11, 2017 at 11:06 AM, Jason Gustafson wrote: > +1 > > On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote: > > > Looks like there was a good consensus on the discuss thread for KIP-106 > so > > lets move to a vote. > > > > Please chime in if you would like to change the default

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Ismael Juma
Thanks for raising this, +1. Ismael On Wed, Jan 11, 2017 at 6:56 PM, Ben Stopford wrote: > Looks like there was a good consensus on the discuss thread for KIP-106 so > lets move to a vote. > > Please chime in if you would like to change the default for > unclean.leader.election.enabled from tru

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Sriram Subramanian
+1 On Wed, Jan 11, 2017 at 11:10 AM, Ismael Juma wrote: > Thanks for raising this, +1. > > Ismael > > On Wed, Jan 11, 2017 at 6:56 PM, Ben Stopford wrote: > > > Looks like there was a good consensus on the discuss thread for KIP-106 > so > > lets move to a vote. > > > > Please chime in if you w

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Grant Henke
+1 On Wed, Jan 11, 2017 at 1:23 PM, Sriram Subramanian wrote: > +1 > > On Wed, Jan 11, 2017 at 11:10 AM, Ismael Juma wrote: > > > Thanks for raising this, +1. > > > > Ismael > > > > On Wed, Jan 11, 2017 at 6:56 PM, Ben Stopford wrote: > > > > > Looks like there was a good consensus on the disc

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Jay Kreps
+1 On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote: > Looks like there was a good consensus on the discuss thread for KIP-106 so > lets move to a vote. > > Please chime in if you would like to change the default for > unclean.leader.election.enabled from true to false. > > https://cwiki.apa

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Ian Wrigley
+1 (non-binding) > On Jan 11, 2017, at 11:33 AM, Jay Kreps wrote: > > +1 > > On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote: > >> Looks like there was a good consensus on the discuss thread for KIP-106 so >> lets move to a vote. >> >> Please chime in if you would like to change the def

[ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Gwen Shapira
The PMC for Apache Kafka has invited Grant Henke to join as a committer and we are pleased to announce that he has accepted! Grant contributed 88 patches, 90 code reviews, countless great comments on discussions, a much-needed cleanup to our protocol and the on-going and critical work on the Admin

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Sriram Subramanian
Congratulations Grant! On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira wrote: > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews, countless great > comments on discussio

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Vahid S Hashemian
Congrats Grant! --Vahid From: Sriram Subramanian To: users@kafka.apache.org Cc: d...@kafka.apache.org, priv...@kafka.apache.org Date: 01/11/2017 11:58 AM Subject:Re: [ANNOUNCE] New committer: Grant Henke Congratulations Grant! On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shap

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Jason Gustafson
Congrats! On Wed, Jan 11, 2017 at 11:57 AM, Sriram Subramanian wrote: > Congratulations Grant! > > On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira wrote: > > > The PMC for Apache Kafka has invited Grant Henke to join as a > > committer and we are pleased to announce that he has accepted! > > > >

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Ismael Juma
Congratulations Grant, well deserved. :) Ismael On 11 Jan 2017 7:51 pm, "Gwen Shapira" wrote: > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews, countless great > com

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Ben Stopford
Congrats Grant!! On Wed, 11 Jan 2017 at 20:01, Ismael Juma wrote: > Congratulations Grant, well deserved. :) > > Ismael > > On 11 Jan 2017 7:51 pm, "Gwen Shapira" wrote: > > > The PMC for Apache Kafka has invited Grant Henke to join as a > > committer and we are pleased to announce that he has a

Re: [VOTE] KIP-106 - Default unclean.leader.election.enabled True => False

2017-01-11 Thread Jeff Widman
+1 nonbinding. We were bit by this in a production environment. On Wed, Jan 11, 2017 at 11:42 AM, Ian Wrigley wrote: > +1 (non-binding) > > > On Jan 11, 2017, at 11:33 AM, Jay Kreps wrote: > > > > +1 > > > > On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote: > > > >> Looks like there was a

Re: Is there a performance problem with new broker + old log.message.format.version + new consumer?

2017-01-11 Thread Ismael Juma
Hi Jeff, The new consumer also supports the old message format without requiring conversion. Ismael On 11 Jan 2017 6:52 pm, "Jeff Widman" wrote: > We upgraded our Kafka clusters from 0.8.2.1 to 0.10.0.1, but most of our > consumers use older libraries that do not support the new message format

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Eno Thereska
Congrats! Eno > On 11 Jan 2017, at 20:06, Ben Stopford wrote: > > Congrats Grant!! > On Wed, 11 Jan 2017 at 20:01, Ismael Juma wrote: > >> Congratulations Grant, well deserved. :) >> >> Ismael >> >> On 11 Jan 2017 7:51 pm, "Gwen Shapira" wrote: >> >>> The PMC for Apache Kafka has invited G

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Guozhang Wang
Congratulations Grant!! On Wed, Jan 11, 2017 at 12:06 PM, Ben Stopford wrote: > Congrats Grant!! > On Wed, 11 Jan 2017 at 20:01, Ismael Juma wrote: > > > Congratulations Grant, well deserved. :) > > > > Ismael > > > > On 11 Jan 2017 7:51 pm, "Gwen Shapira" wrote: > > > > > The PMC for Apache K

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Mickael Maison
Congratulations Grant ! On Wed, Jan 11, 2017 at 8:29 PM, Guozhang Wang wrote: > Congratulations Grant!! > > On Wed, Jan 11, 2017 at 12:06 PM, Ben Stopford wrote: > >> Congrats Grant!! >> On Wed, 11 Jan 2017 at 20:01, Ismael Juma wrote: >> >> > Congratulations Grant, well deserved. :) >> > >> >

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Matthias J. Sax
Congrats! On 1/11/17 12:52 PM, Mickael Maison wrote: > Congratulations Grant ! > > On Wed, Jan 11, 2017 at 8:29 PM, Guozhang Wang wrote: >> Congratulations Grant!! >> >> On Wed, Jan 11, 2017 at 12:06 PM, Ben Stopford wrote: >> >>> Congrats Grant!! >>> On Wed, 11 Jan 2017 at 20:01, Ismael Juma

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Jay Kreps
Congrats Grant! -Jay On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira wrote: > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews, countless great > comments on discussion

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Kaufman Ng
Congrats Grant! On Wed, Jan 11, 2017 at 4:28 PM, Jay Kreps wrote: > Congrats Grant! > > -Jay > > On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira wrote: > > > The PMC for Apache Kafka has invited Grant Henke to join as a > > committer and we are pleased to announce that he has accepted! > > > > G

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Becket Qin
Congrats Grant! On Wed, Jan 11, 2017 at 2:17 PM, Kaufman Ng wrote: > Congrats Grant! > > On Wed, Jan 11, 2017 at 4:28 PM, Jay Kreps wrote: > > > Congrats Grant! > > > > -Jay > > > > On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira > wrote: > > > > > The PMC for Apache Kafka has invited Grant Hen

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Mayuresh Gharat
Congrats Grant ! Thanks, Mayuresh On Thu, Jan 12, 2017 at 3:47 AM, Kaufman Ng wrote: > Congrats Grant! > > On Wed, Jan 11, 2017 at 4:28 PM, Jay Kreps wrote: > > > Congrats Grant! > > > > -Jay > > > > On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira > wrote: > > > > > The PMC for Apache Kafka h

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread James Cheng
Congrats, Grant!! -James > On Jan 11, 2017, at 11:51 AM, Gwen Shapira wrote: > > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews, countless great > comments on discu

java.lang.OutOfMemoryError: Java heap space while running kafka-consumer-perf-test.sh

2017-01-11 Thread Check Peck
I am trying to run kafka performance script on my linux box. Whenever I run "kafka-consumer-perf-test.sh", I always get an error. In the same box, I am running "kafka-producer-perf-test.sh" as well and that is not failing at all. Looks like something is wrong with "kafka-consumer-perf-test.sh". I

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Martin Gainty
viel gluck Herr Henke! Martin __ From: James Cheng Sent: Wednesday, January 11, 2017 5:37 PM To: users@kafka.apache.org Cc: d...@kafka.apache.org; priv...@kafka.apache.org Subject: Re: [ANNOUNCE] New committer: Grant

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Jun Rao
Grant, Thanks for all your contribution! Congratulations! Jun On Wed, Jan 11, 2017 at 2:51 PM, Gwen Shapira wrote: > The PMC for Apache Kafka has invited Grant Henke to join as a > committer and we are pleased to announce that he has accepted! > > Grant contributed 88 patches, 90 code reviews,

Re: Apache Kafka integration using Apache Camel

2017-01-11 Thread Asaf Mesika
Don't specify Kafka dependencies. Camel will transitively bring it. Otherwise you are causing version conflict. On Mon, 9 Jan 2017 at 14:20 Kamal C wrote: > Can you enable DEBUG logs ? It'll be helpful to debug. > > -- Kamal > > On Mon, Jan 9, 2017 at 5:37 AM, Gupta, Swati wrote: > > > Hello All