Re: Kafka issue (Leaders not elected)

2020-11-23 Thread Liam Clarke-Hutchinson
Hi Mohammad, Leader elections are generally seldom and usually quick. If a leader is unable to elected, this could point to a) replication being set too low or b) inter-broker communication issues, plus a bunch of other possible problems. You'd have to look into your Kafka broker's logs to figur

Re: kafka issue

2019-12-10 Thread Valentin Forst
> > Am 10.12.2019 um 06:36 schrieb Valentin : > > Hi Chao, > > I suppose, you would like to know: > within a consumer group which message is coming from which partition, since > partitions corresponds to broker and broker = ip, right? > > Well, if you really want to know this, then you have to

Re: kafka issue

2019-12-09 Thread Valentin
Hi Chao, I suppose, you would like to know: within a consumer group which message is coming from which partition, since partitions corresponds to broker and broker = ip, right? Well, if you really want to know this, then you have to get the context. E.g within a processor there is a method call

Re: Kafka issue

2018-08-19 Thread Nan Xu
maybe I should highlight, I only publish 1 key. so only one broker is going to handle it. and only 1 stream instance handle it too. what's the typical throughput/latency I should expect in this case? assuming the processing logic is very very simple, just get data(integer) and sum. I am more expect

Re: Kafka issue

2018-08-19 Thread Nan Xu
I did several test. one is with 10 brokers (remote server), one with 3 brokers. (local docker) both exhibit the same behavior, I was thinking the same but from at least the kafka log, I don't see a rebalance happening. and I am sure my cpu is only used about half. and all broker still running.

Re: Kafka issue

2018-08-19 Thread Shantanu Deshmukh
How many brokers are there in your cluster? This error usually comes when one of the brokers who is leader for a partition dies and you are trying to access it. On Fri, Aug 17, 2018 at 9:23 PM Harish K wrote: > Hi, >I have installed Kafka and created topic but while data ingestion i get > so

Re: Kafka Issue [Corrupted broker]

2014-12-09 Thread Joe Stein
It looks like broker 5 is in a bad state. You are likely going to have to shut it down. From there you have a few options and depending on your environment setup will dictate if you do shut it down and/or what you do after that. Spinning up another server with broker.id == 5 and let replication he

Re: Kafka issue with "Reconnect due to socket error"

2013-07-09 Thread Jun Rao
The reconnect is due to the socket connection to Kafka broker. Technically, reconnect worked. So the tool still produces the output. The reconnection is weird though. Do you see the same reconnection when you run console consumer? Thanks, Jun On Tue, Jul 9, 2013 at 11:06 AM, Dennis Haller wrote

Re: Kafka issue with "Reconnect due to socket error"

2013-07-09 Thread Dennis Haller
Hi, Thanks for your reply. I can't see anything in the broker logs, , except this keeps coming up in the zookeeper log (instance 1 out of 3): WARN [SyncThread:1:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:1 took 1298ms which will adversely effect operation latency. See the ZooK

Re: Kafka issue with "Reconnect due to socket error"

2013-07-08 Thread Jun Rao
The consumer reconnects because the broker closed the socket. Any error/exception on the broker side around the same time? Thanks, Jun On Mon, Jul 8, 2013 at 10:25 AM, Dennis Haller wrote: > I have a 4-broker Kafka system running in Amazon EC2, and we are using > Kafka 0.8 beta1. Most of the