Re: Problem with timestamp in Producer

2016-10-30 Thread Debasish Ghosh
Thanks Matthias again for all the suggestions .. On Monday 31 October 2016, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Just one comment to add: > > > it produces messages based > >> on 0.9.0 which obviously cannot be consumed by 0.10.0 brokers. > > Strictly sp

Re: Problem with timestamp in Producer

2016-10-30 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Just one comment to add: > it produces messages based >> on 0.9.0 which obviously cannot be consumed by 0.10.0 brokers. Strictly speaking, this is not correct. Brokers are compatible to older versions of clients (not the other way round though). Fo

Re: Kafka client circuit breaker

2016-10-30 Thread Tauzell, Dave
Once enough failures happen the circuit is marked open. The client would then periodically try some messages until it works again. Others would be failed. There are a number of existing circuit breaker libraries you can use in the meantime like the Netflix one. Dave > On Oct 30, 2016, at 20:

Re: Kafka client circuit breaker

2016-10-30 Thread R Krishna
So, basically, you want the clients to gracefully handle timeouts and short circuit if it is unavailable broker exception and also clients should be able automatically recover when it is up? What about the messages that come in when broker is not available? On Sun, Oct 30, 2016 at 12:25 PM, Andrey

connecting to multiple kafka clusters with kerberos enabled

2016-10-30 Thread Digumarthi, Prabhakar Venkata Surya
We need to use the following configuration to connect to kafka cluster , I should I configure a single client to connect to multiple clusters with different JAAS configuration. Is it possible to do that ? KafkaClient { com.sun.security.auth.module.Krb5LoginModule required u

Re: Kafka client circuit breaker

2016-10-30 Thread Andrey Dyachkov
R Krishna, We have default timeout for producer and consumer it is 30 secs. If Kafka is not available we will wait 30 secs on each request. Decreasing timeout won't help much because we have a lot requests but circuiting them will help us tremendous. We could implement such logic before actually c

Kafka Streams fails permanently when used with an unstable network

2016-10-30 Thread Frank Lyaruu
I have a remote Kafka cluster, to which I connect using a VPN and a not-so-great WiFi network. That means that sometimes the Kafka Client loses briefly loses connectivity. When it regains a connection after a while, I see: org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be c

Re: Problem with timestamp in Producer

2016-10-30 Thread Debasish Ghosh
I think I found out what happened .. I was installing Kafka under DC/OS on AWS and following this doc https://dcos.io/docs/1.8/usage/tutorials/kafka/ for kafka installation. This works fine and installs kafka version 0.10.0. But in the example where it shows how to produce and consume messages, th

monitor replica lag

2016-10-30 Thread Ritesh Tijoriwala
Hi All, We are using kafka 0.8.1.1 and one of our brokers got kicked out of the ISR list after a restart. The restart was to increase the number of replica fetcher threads from 1 to 4. Now, it fails to get back in the ISR list even after waiting for 1 day. We had it restart few times but no luc

Re: Problem with timestamp in Producer

2016-10-30 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 The simplest way should be to check the java classpath. Insert an echo $CLASSPATH at the end of bin/kafka-run-class.sh Than run bin/kafka-console-producer.sh with no argument. You should see the classpath be printed out. Look for 'kafka-clients-