Facing large election time in Kafka

Hi, We are currently facing a prolonged leader election time, approx 2 mins, in a Kafka cluster (version 2.8.2) that is configured with Zookeeper. This cluster has large number of topic partitions. The issue arises during the rolling restarts of the servers in the Kafka cluster. This extended lea

Re: producer fast fail

Is there a way for our producer to fail quickly if not available? No. Client will just try to re-connect internally. They don't rise any error if they disconnect. On 8/22/24 5:45 AM, Keith Brown wrote: Sometimes out is unavailable. Our producer gives this error [kafka-producer-network-th

Re: java kafka multithreaded

I don't think there is any example, but seems to be more or less "straightforward" to do? However, I am wondering about the usage pattern? Does not sound like a good fit for Kafka. Producers and consumer should be long lived. It's an anti pattern to use short lived clients (Kafka is optimized

Re: sending and receiving message

What do you mean by "timeout a consumer"? Given that a consumer calls `poll()` and `poll()` might also return with no data, the consumer code is in full control anyway... I want to make sure a certain message gets pushed if not then exit. This sounds like writing. How it it related to a co

Re: MirrorMaker 2 : Offset issues since 3.5.x+

Hi Tom, Thanks for your question! In Kafka 3.5 there was an overhaul of the offset translation logic to avoid a data loss bug. This [1] is the ticket that I've been using to represent that overhaul. In the implementation for that fix, we traded a lot of availability (read: consumer lag) for corre

MirrorMaker 2 : Offset issues since 3.5.x+

We’ve been using MM2 one way for Active/Passive clusters for several years now. We started running into issues with 3.5.1. It hasn’t been keeping up with with consumer offset like it used to. To test this out we’ve done rollbacks to 3.4.0 and the offset issue corrects itself. Looking at the issu

RE: Re: Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

Hi Ömer Many thanks for your response. The bandwidth between the two sites is many orders of magnitude greater than the 1.5 MB throughput I’m able to achieve, so I’m confident that the issue isn’t a case of low bandwidth. I have used the -1 setting on the broker (as well as setting explicitly

Re: Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

Hi Austin, I think it is worth to mention also the network bandwidth without even Kafka involved. You can create a test bench with basic Linux toolkit like ncat. Tuning socket buffers hardly make sense if the bandwidth is already low between the producer and remote site. When OS is configured t

Kafka seems to ignore socket.receive.buffer.bytes and socket.send.buffer.bytes bytes

Hi List I am running Kafka 3.7.1 on CentOS 7.9. I have written a Kafka producer program in Python using confluent_kafka (which uses librdkafka). When the program is running on a machine in the same data centre as the Kafka cluster, a single producer writing 1,000 byte messages to a single topi

producer fast fail

Sometimes out is unavailable. Our producer gives this error [kafka-producer-network-thread | producer-1] WARN org.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Bootstrap broker 127.0.0.1:9091 (id: -1 rack: null) disconnected [main] INFO org.apache.kafka.clients.producer.Kaf