How Kafka Manages Network Partition

Can anyone clarify how Kafka manages below network partition? Say we have this configuration before the network partition, - Kafka cluster has three brokers (say broker_0, broker_1, broker2). - broker_1 is the leader. - ISR has been reduced to broker_1 and broker_2. - There is a Zooke

Re: How Kafka Manages Network Partition

AFAIK there won't be two leaders. Once brokers lose connection with ZooKeeper, a new leader will be elected (whichever can still access ZooKeeper) and the remaining brokers will fall behind in replication. Now depending on your config, there might be several other issues: not enough replicas to sa

Re: Purging dead consumer ids from _consumer_offsets?

I'm also very interested in this question - any update on this? thanks! Marina Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Thursday, September 5, 2019 6:30 PM, Ash G wrote: > _consumer_offsets is becoming rather big > 1 TB. Is there a way to purge > dead/inactive c

Shipping Kafka logs

Hello Can you please point me to the doc or guide me on shipping kafka logs to elk. I understand it is a little different than how we ship other logs to elk. Kindly help me with this Thanks

Re: Purging dead consumer ids from _consumer_offsets?

+ Following the post. On Mon, Sep 23, 2019 at 6:31 PM Marina Popova wrote: > I'm also very interested in this question - any update on this? > thanks! > Marina > > > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Thursday, September 5, 2019 6:30 PM, Ash G > wrote

Re: Shipping Kafka logs

Hi Eva, Hope this helps. https://www.elastic.co/guide/en/logstash/current/plugins-inputs-kafka.html Kind regards, Liam Clarke On Tue, Sep 24, 2019 at 2:57 AM Eva Sheeva wrote: > Hello > > Can you please point me to the doc or guide me on shipping kafka logs to > elk. I understand it is a litt

Shipping Kafka logs

You can also try https://docs.confluent.io/current/connect/kafka-connect-elasticsearch/index.html I think both links are about shipping kafka messages to elk, not actual kafka application logs -Original Message- From: Liam Clarke Sent: Monday, September 23, 2019 5:21 PM To: users@kafk

Re: How Kafka Manages Network Partition

Ok. If min.insync.replicas is 1 and the acknowledgment method is acks=1, - Zookeeper thinks that broker_1 and broker_2 are dead and elect broker_0 as the new leader. - But, broker_2 can still send fetch requests to broker_1 and original ISR (broker_1, broker_2) will remain the same.