Re: [ANNOUNCE] New committer: Ismael Juma

Thank you Neha and the PMC for the opportunity. I am both grateful and excited. :) And thanks to everyone else in the Kafka community too. Ismael On Mon, Apr 25, 2016 at 10:52 PM, Neha Narkhede wrote: > The PMC for Apache Kafka has invited Ismael Juma to join as a committer and > we are please

Re: Consumer doesn't run delayed tasks while under load

Hi Robert, I think the scenario you describe is possible. It would be nice if you can provide more information on how to reproduce the issue. Also, would you mind to file a JIRA to track the issue? In the JIRA, it would be nice if you can share the configurations of the consumer. Thanks, Liquan

0.9.0.1 High CPU usage on broker - Why is the default heart beat interval set too low (3 seconds)?

We have been investigating an unreasonably high CPU usage of the Kafka process when there's no _real_ activity going on between the consumers and the broker. We had this issue in 0.8.x days and is exactly the same as what's being tracked in this JIRA https://issues.apache.org/jira/browse/KAFKA-

Re: [ANNOUNCE] New committer: Ismael Juma

Congrats Ismael On Tue, Apr 26, 2016 at 12:58 PM, Ismael Juma wrote: > Thank you Neha and the PMC for the opportunity. I am both grateful and > excited. :) > > And thanks to everyone else in the Kafka community too. > > Ismael > > On Mon, Apr 25, 2016 at 10:52 PM, Neha Narkhede wrote: > > > The

Re: electing leader failed and result in 0 latest offset

Hi Qi, Just to confirm, as you seeing the offset reset to 0 in the new consumer? I am not sure the root cause of the leader election failure. But as the new Kafka consumer is storing the offset in Kafka, it is possible to reset the offset to 0 for some topic partitions when the leader for a offse

Re: 0.9.0.1 High CPU usage on broker - Why is the default heart beat interval set too low (3 seconds)?

Hi Jaikiran, Thanks for the email and the detailed analysis. One reason for setting the heartbeat interval to a lower value is for faster failure detection. On every received heartbeat, the coordinator starts (or resets) a timer. If no heartbeat is received when the timer expires, the coordinator

RE: Detecting rebalance while processing ConsumerRecords (0.9.0.1)

Hi Vinay, "Regarding identifying a rebalance, how about comparing array used for consumer pause with current assignments of consumer?" I did consider checking that in the commitSync exception handler but didn't try it because if this is in the consumer that has caused the rebalance (i.e. the o

[GroupCoordinatorNotAvailableException] consumer error on poll request

Hello, I am using KafkaProducer and KafkaConsumer, version 0.90, in java, the producer works good (the Console Consumer provided in samples receives the producer data). But in the configured KafkaConsumer I got the /org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException/**error

Rebalance events when auto commit is disabled in 0.9 consumer

Hi, Per consumer group we have many consumers that aggregate data for a predefined amount of time and then write the aggregated data to a db. We're looking to set enable.auto.commit to false and write the offsets only when the aggregated data has been written to db. In that context we're not su

RE: Detecting rebalance while processing ConsumerRecords (0.9.0.1)

Hi Phil, CommitSync sends a heartbeat request on each call but it seems that somehow it stops sending a heartbeat request after a meta refresh till next poll. I asked about this on dev list and came to know that this is fixed in 0.10.0.0 which is next version. I heve not gone in to detail of defec

Re: Compaction does not seem to kick in

Yes, we have indeed set the cleanup.policy=compact for the specific topic. On 4/21/16, 11:58 PM, "Manikumar Reddy" wrote: >Did you set broker config property log.cleanup.policy=compact or topic >level property cleanup.policy=compact ? > >On Thu, Apr 21, 2016 at 7:16 PM, Kasim Doctor wrote: >

Re: [ANNOUNCE] New committer: Ismael Juma

Congratulations, very well deserved. On Apr 25, 2016 10:53 PM, "Neha Narkhede" wrote: > The PMC for Apache Kafka has invited Ismael Juma to join as a committer and > we are pleased to announce that he has accepted! > > Ismael has contributed 121 commits >

Re: [ANNOUNCE] New committer: Ismael Juma

Congratulations! On Tue, Apr 26, 2016 at 6:57 AM, Gwen Shapira wrote: > Congratulations, very well deserved. > On Apr 25, 2016 10:53 PM, "Neha Narkhede" wrote: > > > The PMC for Apache Kafka has invited Ismael Juma to join as a committer > and > > we are pleased to announce that he has accepted

Re: [ANNOUNCE] New committer: Ismael Juma

Congrats Ismael, very well deserved!! On Tue, Apr 26, 2016 at 7:16 AM -0700, "Sriram Subramanian" wrote: Congratulations! On Tue, Apr 26, 2016 at 6:57 AM, Gwen Shapira wrote: > Congratulations, very well deserved. > On Apr 25, 2016 10:53 PM, "Neha Narkhede" wrote: > > > The PMC for

Filter plugins in Kafka

Hi, Do we have any plugin which is available, which can be used as a filtering mechanism on the data it's working on? Regards, Subramanian. K

RE: Detecting rebalance while processing ConsumerRecords (0.9.0.1)

Hi Vinay, Thanks for that information, it's good to know that will be fixed but I'm not sure what would trigger this to happen in the 0.9.0.1 release? What would cause the metadata refresh to be called while the consumer is processing a batch of messages where I'm committing after each message

Re: Detecting rebalance while processing ConsumerRecords (0.9.0.1)

try setting props.put(ConsumerConfig.METADATA_MAX_AGE_CONFIG, 4); default is 30 ms On Tue, Apr 26, 2016 at 11:34 AM, Phil Luckhurst wrote: > Hi Vinay, > > Thanks for that information, it's good to know that will be fixed but I'm > not sure what would trigger this to happen in the 0.9.0

Producer and consumer awareness after adding partitions

Hi I am doing some tests to understand how kafka behaves when adding partitions to a topic while producing and consuming. My test is like this I launch 3 brokers I create a topic with 3 partitions and replication factor = 2 $ ./bin/kafka-topics.sh --zookeeper zookeeper-1 --create --topic topic1

Re: Filter plugins in Kafka

Hi Subramanian, Could you elaborate a bit more on "filtering"? Do you want to read raw data from Kafka, and send the filtered data back to Kafka as a separate topic, or do you want to read raw data from an external service and send the filtered data into Kafka? Guozhang On Tue, Apr 26, 2016 at

Re: Metadata Request Loop?

Hello. I found the issue. The Ops team deployed kafka 0.8.1 and all my code was 0.9.0. Simple mistake and one that I should have thought of sooner. Once I had them bump up to the latest kafka all was well. Thank you for your help! v > On Apr 25, 2016, at 2:54 PM, vinay sharma wrote: > > Hi,

Re: Detecting rebalance while processing ConsumerRecords (0.9.0.1)

Hi Phil, Config ConsumerConfig.METADATA_MAX_AGE_CONFIG has default 30 ms. This config drives a mechanism where a proactive meta data refresh request is issued by consumer periodically. i have seen that i get log about successful heartbeat along with commit only before this request. once this r

Re: Filter plugins in Kafka

Instantly reminded me of Streams API, where you can use Java8 streams semantics (filter being one of them) to do the first thing in Gouzhang's response (filter messages from one topic into another - I assume that's what you were looking for). Marko Bonaći Monitoring | Alerting | Anomaly Detection

Re: 0.9.0.1 High CPU usage on broker - Why is the default heart beat interval set too low (3 seconds)?

Thanks for responding, Liquan. Just so I understand better, where does the coordinator reside? On the broker side? And the coordinator knows the heart beat interval configured on each consumer that's connected to the broker, I am guessing. For the usecase we have, having a higher value for hear

Re: Encryption at Rest

Another option is to encrypt the data before you hand it to Kafka and have the downstream decrypt it. This takes care of on-disk on on-wire encryption. We did a proof of concept of this: http://www.symantec.com/connect/blogs/end-end-encryption-though-kafka-our-p roof-concept ( http://symc.l

topic=LEADER_NOT_AVAILABLE

Hello, I am trying to run a simple kafka cluster on an OpenNebula cloud. I am using the 0.9.0. Kafka binary. Each time I try to connect to the broker (from localhost or from a different machine), I get this error: [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkCl

auto leader rebalancing

Hello, We have auto.leader.rebalance.enable = True, other options are by default (10% imbalance ratio and 300 seconds). We have a check that reports leadership imbalance: critical: Leadership out of balance for topic mp-auth. Leader counts: { "104224873"=>84, "104224876"=>22, "104224877"=>55, "1

Re: topic=LEADER_NOT_AVAILABLE

Hi Oliver, I am not familiar with OpenNebula cloud, but in AWS the internal private IP address is usually different from its public IP address: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/elastic-ip-addresses-eip.html http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-ip-addressing.

Re: kafka.common.QueueFullException

I am using the send method of *kafka.javaapi.producer.Producer* */** * Use this API to send data to multiple topics * @param messages list of producer data objects that encapsulate the topic, key and message data */def send(messages: java.util.List[KeyedMessage[K,V]]) {import coll

Re: Metadata Request Loop?

I spoke too soon. It's back doing the same thing.. which is really odd. > On Apr 26, 2016, at 12:24 PM, Fumo, Vincent > wrote: > > Hello. I found the issue. The Ops team deployed kafka 0.8.1 and all my code > was 0.9.0. Simple mistake and one that I should have thought of sooner. Once > I ha

Re: Consumer doesn't run delayed tasks while under load

Absolutely; I created https://issues.apache.org/jira/browse/KAFKA-3627 and included fairly detailed steps to reproduce, including the consumer configuration. Thanks! From: Liquan Pei Sent: Tuesday, April 26, 2016 3:51 AM To: users@kafka.apache.org Subje

Re: Partition fetching stalls with 0.9.0 new consumer

You may be hitting https://issues.apache.org/jira/browse/KAFKA-2978, if you're using 0.9.0.0 From: Alex Popiel Sent: Monday, April 25, 2016 7:40 PM To: 'users@kafka.apache.org' Subject: Partition fetching stalls with 0.9.0 new consumer Hello, folks. I'm

Re: Rebalance events when auto commit is disabled in 0.9 consumer

Hi Balthasar, It is possible that after rebalance, the topic partitions assigned to a consumer instance are changed. To prevent state loss, a custom ConsumerRebalanceListener is needed to properly save the state. In the onPartitionRevoked method, you need to trigger the write to database and wait

Re: Producer and consumer awareness after adding partitions

The time is controlled by metadata.max.age.ms On Wed, Apr 27, 2016 at 1:19 AM Luciano Afranllie wrote: > Hi > > I am doing some tests to understand how kafka behaves when adding > partitions to a topic while producing and consuming. > > My test is like this > > I launch 3 brokers > I create a to

Reg slack channel

Hi, Is there a slack channel for discussing issues related to Kafka? I would like to get an invite. Thanks! -- Kanagha

Re: Metadata Request Loop?

What versions of kafka and client API are you using now? On Apr 26, 2016 3:35 PM, "Fumo, Vincent" wrote: > I spoke too soon. It's back doing the same thing.. which is really odd. > > > On Apr 26, 2016, at 12:24 PM, Fumo, Vincent < > vincent_f...@cable.comcast.com> wrote: > > > > Hello. I found th

Re: Reg slack channel

We use apache JIRA and mailing lists for any discussion. Thanks, Harsha On Tue, Apr 26, 2016, at 06:20 PM, Kanagha wrote: > Hi, > > Is there a slack channel for discussing issues related to Kafka? I would > like to get an invite. Thanks! > > > -- > Kanagha

Re: Reg slack channel

There's also an irc channel on freenode #apache-kafka that hosts some periodic user discussion. On Tue, Apr 26, 2016 at 7:17 PM, Harsha wrote: > We use apache JIRA and mailing lists for any discussion. > Thanks, > Harsha > > On Tue, Apr 26, 2016, at 06:20 PM, Kanagha wrote: >> Hi, >> >> Is there