Good day, We're currently running a Kafka cluster in our staging environment, testing everything out and so far it runs fairly smoothly.
One thing that we're currently looking into is scaling a cluster that is already up 'n serving data, and I have some questions regarding moving partitions between brokers. My current interpretation is that if I start a partition reassignment, for the sake of simplicity let's assume it's just for a single partition, the new leader will first become a follower of the current leader, and when it has caught up it'll transfer leadership over to itself? If so, is Kafka only concerned about data that has been flushed to disk? So if there's data that hasn't been flushed to disk yet it'll be discarded when the new leader gains leadership? What happens if neither of the new ARs are part of the previous AR set? I suppose that they'll all be followers and when the new leader has caught up it'll assume leadership? I've played around with controlled shutdown and noticed that the broker that's shutting down is truncating logs prior to actually shutting down, which I don't really understand? I cannot really tell whether it does this for all topics or just a subset of the ones that it's AR for as we have quite a lot of topics, but I figured I'd just throw it out there, as I feel like it could be explained by the answers to the questions above. Should mention that we're running v0.8.1.1 with mostly stock configuration, and that our producers are currently using ACK = 1 as well. Best regards, Mathias
