Hello, all. We're running multiple Kafka clusters in AWS, and thus multiple Zookeeper clusters as well. When we roll out changes to our zookeeper nodes (which involves changes to the AMI, which means terminating the zookeeper instance and bringing up a new one in its place) we have to restart our Kafka brokers one at a time so they can pick up the new zookeeper IP address.
What we've noticed is that, as the brokers are restarted, we get alerts for under-replicated partitions, which seems strange since it seems like the shutdown process should take care of moving any replicas and the leadership election process. This is causing us some pain because it means that we get pages whenever we roll out changes to Zookeeper. Does anybody have any ideas why this would be happening, and how we can avoid it? Thanks. -Jack Lund Braintree Payments
