Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

+1 (binding). On Wed, Jan 17, 2018 at 6:09 PM, Matthias J. Sax wrote: > Hi, > > I would like to start the vote for KIP-247: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 247%3A+Add+public+test+utils+for+Kafka+Streams > > > -Matthias > > -- -- Guozhang

This server is not the leader for that topic-partition

Hi: I use kafka for days. and it's normal at beginning . yesterday. I found kafka topic leader often switch to one to another. and in kafka server.log throw the error： This server is not the leader for that topic-partition. kafka-version: kafka_2.11-1.0.0 funk...@l

Unrecoverable kafka cluster after failed rolling restart

Hi everyone, Fortunately this didn't happen with a production cluster, but here is the sequence of events with this 10 broker cluster: 1) Stop all kafka brokers servers (this was a mistake on my part; I intended to do a rolling restart) 2) Start 5 brokers a few minutes apart each 3) Start the rest

[VOTE] KIP-247: Add public test utils for Kafka Streams

Hi, I would like to start the vote for KIP-247: https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams -Matthias signature.asc Description: OpenPGP digital signature

Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams

Hi, I updated the KIP and added an `OutputVerifier` class. Looking forward to your feedback. I will start the Vote thread in parallel. -Matthias On 1/17/18 11:40 AM, Matthias J. Sax wrote: > Saïd, > > sorry for late reply. Avro records are basically supported. However, > there are no plans

答复: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations, Rajini Sivaram. Very well deserved! 发件人: Konstantine Karantasis 发送时间: 2018年1月18日 6:23 收件人: d...@kafka.apache.org 抄送: users@kafka.apache.org 主题: Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram Congrats Rajini! -Konstantine On Wed, Jan 17,

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations Rajini! On Thu, Jan 18, 2018 at 9:23 AM, Konstantine Karantasis < konstant...@confluent.io> wrote: > Congrats Rajini! > > -Konstantine > > On Wed, Jan 17, 2018 at 2:18 PM, Becket Qin wrote: > > > Congratulations, Rajini! > > > > On Wed, Jan 17, 2018 at 1:52 PM, Ismael Juma wrote

Re: Kafka Replication Factor

if RF=2 and min.insync.replicas=1 (the default) then you shouldn't have offline partitions if 1 of 3 brokers is down. I'd first double check your topic config (and broker defaults) for the one that went offline to verify RF/Min. Be sure to check each partition they can be different! ( kafka-topics

Re: __consumer_offsets too big

There are several bugs in 0.9 around consumer offsets and compaction and log cleaning. The easiest path forward is to upgrade to the latest 0.11.x. We ended up going to somewhat extreme lengths to deal with 100GB+ consumer offsets. When we tested an upgrade we noticed that when it started compac

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congrats Rajini! -Konstantine On Wed, Jan 17, 2018 at 2:18 PM, Becket Qin wrote: > Congratulations, Rajini! > > On Wed, Jan 17, 2018 at 1:52 PM, Ismael Juma wrote: > > > Congratulations Rajini! > > > > On 17 Jan 2018 10:49 am, "Gwen Shapira" wrote: > > > > Dear Kafka Developers, Users and Fan

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations, Rajini! On Wed, Jan 17, 2018 at 1:52 PM, Ismael Juma wrote: > Congratulations Rajini! > > On 17 Jan 2018 10:49 am, "Gwen Shapira" wrote: > > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since then, she remained > active in the com

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congratulations Rajini! On 17 Jan 2018 10:49 am, "Gwen Shapira" wrote: Dear Kafka Developers, Users and Fans, Rajini Sivaram became a committer in April 2017. Since then, she remained active in the community and contributed major patches, reviews and KIP discussions. I am glad to announce that

Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams

Saïd, sorry for late reply. Avro records are basically supported. However, there are no plans atm to integrate the test driver with a schema registry. Note, that Apache Kafka does not provide a schema registry by itself. If we integrate one RS, we need to integrate all of them -- that is something

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congrats Rajini!! -Matthias On 1/17/18 10:54 AM, Vahid S Hashemian wrote: > Great news! Congratulations Rajini! > > --Vahid > > > > From: Gwen Shapira > To: "d...@kafka.apache.org" , Users > > Date: 01/17/2018 10:49 AM > Subject:[ANNOUNCE] New Kafka PMC Member: Rajini Sivar

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Great news! Congratulations Rajini! --Vahid From: Gwen Shapira To: "d...@kafka.apache.org" , Users Date: 01/17/2018 10:49 AM Subject:[ANNOUNCE] New Kafka PMC Member: Rajini Sivaram Dear Kafka Developers, Users and Fans, Rajini Sivaram became a committer in April 2017. Si

Re: Kafka Streams topology does not replay correctly

We will fix this. No worries. And it is basically only a problem for Stream-Table join... All other operators don't "suffer" from the current behavior. I think in the end, we need to do multiple poll() calls and pause partitions to make sure the internal RecordQueue has data for all input partiti

[ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Dear Kafka Developers, Users and Fans, Rajini Sivaram became a committer in April 2017. Since then, she remained active in the community and contributed major patches, reviews and KIP discussions. I am glad to announce that Rajini is now a member of the Apache Kafka PMC. Congratulations, Rajini

Ambiguous entry in docs

Hi. I read (mostly) all documentation top to bottom and happened to stumble upon passage I cannot understand. It states: https://kafka.apache.org/documentation/#impl_consumerregistration "If the consumer creates a message stream using a topic filter, it also registers a watch on changes (new top

Re: Kafka Streams topology does not replay correctly

I have read through the source, following StreamThread -> AssignedStreamTasks -> StreamTask -> PartitionGroup -> RecordQueue. I see, as you said, that everything depends on what records the consumer initially returns from poll. I wonder how this problem might be solved in the bigger picture. Certa

Re: Kafka Replication Factor

whats your config for min.insync.replicas? 2018-01-17 13:37 GMT+01:00 Sameer Kumar : > Hi, > > I have a cluster of 3 Kafka brokers, and replication factor is 2. This > means I can tolerate failure of 1 node without data loss. > > Recently, one of my node crashed and some of my partitions went off

Re: I would like to increase the bandwidth by binding each broker to each network card in one machine. Is this feasible?

From: 猪少爷 Sent: Tuesday, January 16, 2018 8:21 PM To: users Subject: I would like to increase the bandwidth by binding each broker to each network card in one machine. Is this feasible? I would like to increase the bandwidth by binding each broker to each netw

Re: Kafka Streams topology does not replay correctly

> That depends what data the consumer fetches and this part is hard to predict. For this reason, you need to buffer multiple records in a store, in case data does not arrive in the order and you need it (between different topics) and later do the processing in the correct order when you got all dat

Kafka Replication Factor

Hi, I have a cluster of 3 Kafka brokers, and replication factor is 2. This means I can tolerate failure of 1 node without data loss. Recently, one of my node crashed and some of my partitions went offline. I am not sure if this should be the case. Am I missing something. -Sameer.

Re: [Production Guideline] Is it good idea to provide limited number of brokers(3 brokers) in kafka client in large cluster

Hi Pritam, It is basically about the probabilities, right? If you use 3 brokers as bootstrap servers, what is the probability that all three will be unavailable at the same time? To make it more reliable: - the more brokers in the bootstrap list the lower the probability that all of them will be d

Re: [DISCUSS] KIP-247: Add public test utils for Kafka Streams

Matthias, What about testing topology that use avro schema ? Have you read my previous response ? Thanks. On Wed, Jan 17, 2018 at 3:34 AM Matthias J. Sax wrote: > Colin, > > the TopologyTestDriver does not connect to any broker and simulates > processing of single-partitioned input topics pure

[Production Guideline] Is it good idea to provide limited number of brokers(3 brokers) in kafka client in large cluster

Hi, I have a use case where my kafka cluster will have 1000 brokers and I am writing kafka client. In order to write client, i need to provide brokers list. Question is, what are the recommended guidelines to provide brokers list in client? *Is there any proxy like service available in kafka whi