When doing rolling restarts, I always wait until the under-replicated partition count returns to zero before restarting the next broker. This state is achieved AFTER the last restarted broker returns to a running state. If you just wait for the running state, you risk restarting the next broker before all partitions have returned to healthy, and then you’ll have offline partitions because your minISR is 2.
-- Peter Bukowinski > On Mar 6, 2023, at 7:04 AM, Luis Alves <lmtjal...@gmail.com> wrote: > > Hello, > > I'm doing some tests with rolling restarts in a Kafka cluster and I have a > couple of questions related to the impact of rolling restarts on Kafka > consumers/producers and on the overall process. > > First, some context on my setup: > > - Kafka cluster with 3 nodes. > - Topic replication factor of 3 with minISR of 2. > - All topics have a single partition (I intend to increase the > partitioning factor in the future, but for now it's just 1 for testing > purposes). > - Kafka version is 3.2.3. > - I have two systems that communicate via these Kafka topics. The > high-level flow is: > 1. System A sends a message to a Kafka topic (at a rate of ~10 > events/sec). > 2. System B consumes the message. > 3. System B sends a reply to a Kafka topic. > 4. System A consumes the reply. > - When the system is stable, I see end-to-end latencies (measured on > System A) around 10ms in the 99th percentile. > - System A is using Kafka client 3.3.1, and System B is using Kafka > client 3.4.0. > - Kafka consumers and producers on both systems are with the default > configurations, except that the Kafka consumers have auto-commits disabled. > - All Kafka brokers are configured with controlled.shutdown.enable set > to true. > - The Kafka cluster is running in Kubernetes and deployed using Strimzi > (this is just for awareness). > - The rolling restart process is the following (when using Strimzi to > manage it, and when we try to do it manually): > 1. Restart each broker, one at a time, by sending a SIGTERM to the > broker process. The controller broker is the last one to be restarted. > 2. Only restart the next broker when the current broker reports the > broker state as RUNNING. Note: when we do this manually (without > Strimzi), > we wait to see the end-to-end latencies stabilize before moving > to the next > broker. > > Now, my questions: > > 1. When we do this process with Strimzi (waits for the broker state to > be RUNNING before moving to the next one), we've seen end-to-end latencies > growing up to 1-2 minutes (System A is not even able to send events to the > Kafka topic). This is unexpected because AFAIK the configurations that we > are using are the ones recommended for high availability during rolling > restarts. My question is: is it enough to wait for the broker state to be > RUNNING to move on to the next broker? > 2. When we do this process manually (we wait for end-to-end latencies to > stabilize and only then move to the next broker), we've seen end-to-end > latencies growing up to 1 second. While this is much better than what we > see in 1., my question is whether this latency increase is expected or not. > > Thanks in advance, > Luís Alves
