how client know the server's topology

2021-12-20 Thread yonghua
Hello, I am seeing this doc: https://github.com/zendesk/ruby-kafka who says: This behavior is controlled by the required_acks option to #producer and #async_producer: # This is the default: all replicas must acknowledge. producer = kafka.producer(required_acks: :all) My question

Re: how client know the server's topology

2021-12-20 Thread Weichu Liu
Producer only send the "required_ack=all/1/0" in the request to the partition leader. Leader is responsible to make sure required_ack is fulfilled, then it ack to client as producing succeeded. On Mon, Dec 20, 2021 at 5:23 PM yonghua wrote: > Hello, > > I am seeing this doc: > https://github.co

Enable TLS Authentication + Enable External Access using LoadBalancer

2021-12-20 Thread Wei Yang
Hello, Being new to Kafka, I’d like to deploy a Kafka cluster on K8s with 3 brokers with listenerSecurityProtocolMap: "INTERNAL:SSL,CLIENT:PLAINTEXT,EXTERNAL:SSL" To enable TLS authentication, I use self-signed TLS certificates. To enable external access, for Kafka, it needs to use 3 LoadBalanc

Improving I/O on KafkaStreams

2021-12-20 Thread Murilo Tavares
Hi I have a KafkaStreams application with a reasonably complex, stateful topology. By monitoring it, we can say for sure that it is bounded by writing I/O. This has become way worse after we upgraded KafkaStreams from 2.4 to 2.8. (even though we disabled warm-up replicas by setting "acceptable.reco

Re: Improving I/O on KafkaStreams

2021-12-20 Thread Bruno Cadonna
Hi Murilo, Have you checked out the following blog post on tuning performance of RocksDB state stores [1] especially the section on high disk I/O and write stalls [2]? Do you manage the off-heap memory used by RocksDB as described in the Streams docs [3]? I do not know what may have caused

Re: Facing split brain type of behaviour in Kafka Connect Mirrormaker

2021-12-20 Thread Lehar Jain
Can anyone pls provide some feedback. I don't have logs now but just wanted to confirm how can this case arise and how to ensure that it does not happen again? On Fri, Nov 26, 2021 at 5:44 PM Lehar Jain wrote: > Hello community, > > Recently my team faced an issue with our Kafka Connect Mirrorma