Re: Confluent Kafka 3.2.2 - rebalancing not happenning

Hello, here is the update .. when i ran script - kafka-preferred-replica-election, it did the re-election as required. ./bin/kafka-preferred-replica-election --zookeeper localhost:3181 so does that mean that i need to run the script -> ./bin/confluent-rebalancer to rebalance the data, but for

Re: Adding new user to the broker dynamically

Hello Manikumar, I appreciate your advice , thank you. I tried to use SASL_PLAINTEXT with SCRAM enabled hoping that lack of SSL will help debugging (will switch to SASL_SSL later). I have 3 brokers running on one box with different ports listeners = SASL_PLAINTEXT://:9092 listeners = SASL_PLAINTE

Re: Schema Registry Maven Artifacts Missing?

U can try this .. "io.confluent" % "kafka-avro-serializer"% "3.2.2" regards. On Wed, Aug 2, 2017 at 9:08 AM, Chaoran Yu wrote: > Does anyone know what artifacts I need to include in my project in order > to use Schema Registry? > > I looked at this SO link: https://stackov

Schema Registry Maven Artifacts Missing?

Does anyone know what artifacts I need to include in my project in order to use Schema Registry? I looked at this SO link: https://stackoverflow.com/questions/37317567/how-to-use-the-avro-serializer-with-schema-registry-from-a-kafka-connect-sourcet

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Hi all, after some further discussions, the best thing to show my Idea of how it should evolve would be a bigger mock/interface description. The goal is to reduce the store maintaining processors to only the Aggregators + and KTableSource. While having KTableSource optionally materialized. I

Confluent Kafka 3.2.2 - rebalancing not happenning

Hi All - i'm trying to rebalance Kafka topic (refer link -> http://docs.confluent.io/current/kafka/rebalancer/rebalancer.html), and somehow the rebalancing is not working. Here is what i'm doing -> - i've 4 Kafka brokers & i've made changes to the server.properties file to enable Confluent Metric

Re: Need clarification on Kafka Usage within our product..

Don't know if it helps but says at the bottom " The contents of this website are © 2016 Apache Software Foundation under the terms of the Apache License v2. Apache Kafka, Kafka, and the Kafka logo are either registered trademarks or trademarks of The Apache Software Fou

Re: Kafka Streams Application crashing on Rebalance

> On Aug 1, 2017, at 10:01 AM, Damian Guy wrote: > > Hi, Yes the issue is in 0.10.2 also. Hi, Any chance of a backport to 0.10.2.x? We have lots and lots of state stores. :)

Re: Kafka Streams Application crashing on Rebalance

I'll work on moving to 0.11. Thanks On Tue, Aug 1, 2017, 1:02 PM Damian Guy wrote: > Hi, Yes the issue is in 0.10.2 also. > > On Tue, 1 Aug 2017 at 17:37 Eric Lalonde wrote: > > > > > > On Aug 1, 2017, at 8:00 AM, Damian Guy wrote: > > > > > > It is a bug in 0.10.2 or lower. It has been fixed

Re: increased response time for OffsetCommit requests

Sorry to keep prodding you with questions, but can you quantify the increase for the ProduceRequest? What is the workload you are testing against: specificallly the batch size, message size, linger time settings of the producers in question? I ask because we benchmarked 0.11.0 against the older 0.

Re: Kafka Streams Application crashing on Rebalance

Hi, Yes the issue is in 0.10.2 also. On Tue, 1 Aug 2017 at 17:37 Eric Lalonde wrote: > > > On Aug 1, 2017, at 8:00 AM, Damian Guy wrote: > > > > It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by > > https://issues.apache.org/jira/browse/KAFKA-4494 > > Hi Damien, the Affects Version i

Re: Kafka Streams Application crashing on Rebalance

> On Aug 1, 2017, at 8:00 AM, Damian Guy wrote: > > It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by > https://issues.apache.org/jira/browse/KAFKA-4494 Hi Damien, the Affects Version is set to 0.10.1.0 in KAFKA-4494. Is the issue in 0.10.2.0 as well?

Re: Monitor all stream consumers for lag

Oh that makes a lot of sense now that I think of it. GlobalStores cannot be part of a group since they have to consume the entire stream per instance. On Tue, Aug 1, 2017 at 10:18 AM, Damian Guy wrote: > Hi Garrett, > > The global state store doesn't use consumer groups and doesn't commit > off

Re: Kafka Streams Application crashing on Rebalance

It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by https://issues.apache.org/jira/browse/KAFKA-4494 On Tue, 1 Aug 2017 at 15:40 Marcus Clendenin wrote: > Hi All, > > > > I have a kafka streams application that is doing a join between a KTable > and a KStream and it seems that after it

Kafka Streams Application crashing on Rebalance

Hi All, I have a kafka streams application that is doing a join between a KTable and a KStream and it seems that after it starts loading the KTable if I either restart the application or start a new jar with the same application-id it starts failing. It looks like when it tries to rejoin the app

Re: Getting error while consuming data from broker

Mahesh, Thanks for sharing the info. Is having "Exactly" 8 brokers a "Must" for you? because one of them is technically unnecessary since your cluster can only tolerate 3 failures (even with 7 brokers). Could you please try the following: 1) Stop the cluster. 2) Increase the number of renum.recov

Re: Monitor all stream consumers for lag

Hi Garrett, The global state store doesn't use consumer groups and doesn't commit offsets. The offsets are checkpointed to local disk, so they won't show up with the ConsumerGroupCommand. That said it would be useful to see the lag, so maybe raise a JIRA for it? Thanks, Damian On Tue, 1 Aug 201

Monitor all stream consumers for lag

I have a simple stream setup which reads a source topic and forks to an aggregation with its own statestore, and a flatmap().to("topic1") and that topic is read in to a global state store. I use ConsumerGroupCommand to query for the lag of each consumer on the topics. https://github.com/apache/kaf

Re: Kafka streams store migration - best practices

Thanks for the confirmation. I guess listener would make sense if I did some caching of the store and needed to refresh it for every change in the underlying store. On Tue, Aug 1, 2017 at 6:10 PM, Damian Guy wrote: > No you don't need to set a listener. Was just mentioning as it an option > if y

Re: Kafka streams store migration - best practices

No you don't need to set a listener. Was just mentioning as it an option if you wan't to know that the metadata needs refreshing, On Tue, 1 Aug 2017 at 13:25 Debasish Ghosh wrote: > Regarding the last point, do I need to set up the listener ? > > All I want is to do a query from the store. For t

Why?? Kafka High Level Consumer is 0.8.2.2 Stop Responding After Few Hours... Timeout in Broker Log

I have written a high level Kafka consumer which is not responding on event sent after 2 hours. Only error I see in broker log like this. 2017-08-01 05:29:42,102 INFO kafka.network.Processor: Closing socket connection to /10.202.138.126. 2017-08-01 07:34:24,957 ERROR kafka.network.Proce

Getting error while consuming data from broker

Hi All, We are having 8 broker kafka cluster configured in our setup and created a topic with 8 partitions & 3 replicas. While trying to consume data from one broker(id:6) we are getting below errors and increase in lag for active partition on that host. We even tried restarting, deleting logs a

Re: Kafka streams store migration - best practices

Regarding the last point, do I need to set up the listener ? All I want is to do a query from the store. For that I need to invoke streams.store() first, which can potentially throw an InvalidStateStoreException during rebalancing / migration of stores. If I call streams.store() with retries till

Re: Unit testing changes in core Scala sources

Hi, I'm no gradle expert, but it looks like gradle core:cleanTest core:test would do what you want. Have you looked at the example command lines in the README.md? Cheers, Tom On 27 July 2017 at 15:36, M. Manna wrote: > Hello, > > Please forgive me for asking too simply question (since I haven

Re: increased response time for OffsetCommit requests

Hi Apurva, There are increases in the *Produce* request also. It is not as substantial as compared to *OffsetCommit. *For both of these requests, the major contributor is Remote time. A couple of other metrics that show different behavior post upgrade: 1. *LogStartOffset*: It has drastically de

Re: Kafka streams store migration - best practices

Hi, On Tue, 1 Aug 2017 at 08:34 Debasish Ghosh wrote: > Hi - > > I have a Kafka Streams application that needs to run on multiple instances. > It fetches metadata from all local stores and has an http query layer for > interactive queries. In some cases when I have new instances deployed, > stor

Kafka streams store migration - best practices

Hi - I have a Kafka Streams application that needs to run on multiple instances. It fetches metadata from all local stores and has an http query layer for interactive queries. In some cases when I have new instances deployed, store migration takes place making the current metadata invalid. Here ar