Re: how client know the server's topology

2021-12-20 Thread Weichu Liu
Producer only send the "required_ack=all/1/0" in the request to the partition leader. Leader is responsible to make sure required_ack is fulfilled, then it ack to client as producing succeeded. On Mon, Dec 20, 2021 at 5:23 PM yonghua wrote: > Hello, > > I am seeing this doc: > https://github.co

Re: MessagesOutPerSec JMX metric

2020-04-13 Thread Weichu Liu
This is because of Kafka consumer's fetch model -- a consumer can completely control how the committed offset moves, even resetting them anywhere. Under these circumstances, MessagesOutPerSec is meaningless. On Mon, Apr 13, 2020 at 11:38 AM 张祥 wrote: > Hi, > > I am wondering why there isn't a m

How to alter topic config with broker endpoint

2020-04-13 Thread Weichu Liu
Hi, We have an in-house Kafka cluster and it only exposes the Kafka endpoint (9092) but no Zookeeper endpoint (2181). I wonder if there is a way to change the topic config via the Kafka port rather than zookeeper port? Thanks

Logs for overridden records, when unclean leader election happens

2019-06-13 Thread Weichu Liu
Hi, According to Kafka's replication mechanism, when a unclean leader election happened (suppose unclean.leader.election=true), there are possibilities that replicas of the same partition are inconsistent, i.e. on a certain offset records store different contents. IIUC, Kafka will override one ve

LogCleaner is not removing Transaction Records

2019-05-07 Thread Weichu Liu
Hi, We are using Kafka Streams with exactly-once enabled on a Kafka cluster for a while. Recently we found that the size of __consumer_offsets partitions grew huge. Some partition went over 30G. This caused Kafka to take quite long to load "__consumer_offsets" topic on startup (it loads the topic

Multiple Leaders on single Partition in Kafka Cluster

2018-10-29 Thread Weichu Liu
Hi, We recently saw a split-brain behavior happened in production. There were a controller switch, and then unclean leader elections. It led to 24 partitions (out of 70) had 2 leaders for 2 days until I restarted the broker. Producers and consumers were talking to different "leaders" so the recor

Document fix for configuring SSL

2017-05-22 Thread Weichu Liu
Hi Kafka users, Recently I've been following https://kafka.apache.org/documentation/#security_ssl to configure SSL connections between Kafka and Filebeat (using Sarama library). Basically the doc works perfectly for what is tested against - SSL between Kafka and Kafka-console-producer/consumer. H