Fwd: Monitor the lag for the consumers that are assigned to partitions topic

Hello! I'm working with Kafka 0.9.1 new consumer API. The consumer is manually assigned to a partition. For this consumer I would like to see its progress (meaning the lag). Since I added the group id consumer-tutorial as property, I assumed that I can use the command bin/kafka-consumer-groups.

error: ... protocols are incompatible with those of existing members ??

I am getting this error: Attempt to join group connect-elasticsearch-indexer failed due to: The > group member's supported protocols are incompatible with those of existing > members. This is a single kafka-connect process consuming two topics. The brokers have never changed, and the version of

Re: JVM Optimizations

Barry, No, because Kafka also relies heavily on the OS page cache, which uses memory. You'd roughly want to allocate enough page cache to hold all the messages for your consumers for say, 30s. Kafka also (in our experience on EC2) tends to run out of network far before it runs out of memory or di

Re: JVM Optimizations

Tom, Thanks, that's very good to know. What kind of instances EC2 instances are you using for your brokers? -barry On Fri, Jun 10, 2016 at 4:17 PM, Tom Crayford wrote: > Barry, > > No, because Kafka also relies heavily on the OS page cache, which uses > memory. You'd roughly want to allocate e

Re: error: ... protocols are incompatible with those of existing members ??

I didn't really expect this to help, but still, I tried deleting /all/ topics and recreating them. But still my connect app will no longer run due to this error. Does this error even have anything to do with persisted state, or is the broker complaining about live client connections?

Re: error: ... protocols are incompatible with those of existing members ??

I delete the group using kafka-consumer-groups.sh --delete and still I get the error.

Regex topics in kafka connect?

The kafka connect docs say The Kafka Connect framework manages any changes to the Kafka input, such as > when the set of input topics changes because of a regex subscription. But I can find no other information how to define a wildcard/regex topic subscription. I tried giving the config topic pr

ELB for Kafka

Hi, I am trying to setup Kafka cluster in AWS. Is it possible to have Kafka brokers behind ELB and producers and consumers talk only to ELB ? If not, should we directly expose all brokers to all producers/consumers ? Please advise. Regards, Ram

Re: ELB for Kafka

Kafka itself handles distribution among brokers and which broker consumers and producers connect to. There's no need for an ELB, and you have to directly expose all brokers to producers and consumers. On Friday, 10 June 2016, Ram Chander wrote: > Hi, > > > I am trying to setup Kafka cluster in A

Re: error: ... protocols are incompatible with those of existing members ??

Barry - i believe the error refers to the consumer group "protocol" that is used to decide which partitions get assigned to which consumers. The way it works is that each consumer says it wants to join X group and it can support protocols (1, 2, 3...). The broker looks at all consumers in group X a

KafkaHighLevel consumer in java returns topics which are removed before?

Hi, I have strange problem. I have kafka 9 cluster of 3 nodes. I removed all topics from zookepper, but when I query topics using Java High Level consumer it returns all three removed kafka topics. But how this is possible when I removed them manually, by going into zookeeper shell and removing th

RE: error: ... protocols are incompatible with those of existing members ??

> Date: Fri, 10 Jun 2016 16:54:47 +0530 > Subject: Re: error: ... protocols are incompatible with those of existing > members ?? > From: bkap...@memelet.com > To: users@kafka.apache.org > > I delete the group using kafka-consumer-groups.sh --delete and still I get > the error. MG>Barry...assum

Re: dynamically changing peer role from voting to observer and vice versa

Anyone? Printing e-mails wastes valuable natural resources. Please don't print this message unless it is absolutely necessary. Thank you for thinking green! Sent from my iPhone > On Jun 8, 2016, at 2:31 PM, Nomar Morado wrote: > > Is there any way in the current API to achieve this? > > I a

ELB for Kafka

Hi, Is it possible to have Kafka brokers behind ELB and producers and consumers talk only to ELB ? If not, should we directly expose all brokers to all producers/consumers ? Please advise. Regards, Ram

KStreams.reduceByKey passing nulls to my Deserializer?

Hi, I’m using Kafka Streams (0.10.0) with JRuby, most of my scripts/nodes are working well at this point, except for one which is using reduceByKey. This is the first time I’m trying to use the local state store so it’s possible there’s something misconfigured, I’m not sure. My config is pretty

Re: KStreams.reduceByKey passing nulls to my Deserializer?

> On Jun 10, 2016, at 14:24, Avi Flax wrote: > > Hi, I’m using Kafka Streams (0.10.0) with JRuby, most of my scripts/nodes are > working well at this point, except for one which is using reduceByKey. Whoops, I should probably share my code as well! Here’s the topology: builder.stream(key_se

Re: Question about heterogeneous brokers in a cluster

Thanks Alex and Todd, I really appreciate the insights. Based on what you've shared: spending more time up-front to homogenize the nodes reduces some cognitive load for day-to-day support. Keeping things simple generally wins me over. Thanks again. -Kevin On Thu, Jun 9, 2016 at 10:37 PM, Todd P

Manual update offset for new consumer

When we were using the old consumer, we can use zookeeper client tool to manually set the offset for a consumer group. For the new consumer when the offsets is stored in broker, is there a tool to do the same thing?

Re: KStreams.reduceByKey passing nulls to my Deserializer?

Hello Avi, Yes, this is possible: although we checked nullable keys when doing reduce / aggregations: https://github.com/apache/kafka/blob/0.10.0/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamReduce.java#L67 We do not check if the there are any values returned from the