Thanks, Luke. Both the static and dynamic options for configuring unclean leader election now work in 3.9, thanks to your help.
Dima, if you have only a transient need to do an unclean leader election, using the kafka-leader-election.sh script is still the best way. It avoids the risk for forgetting to turn off unclean leader election afterwards, and it's faster. You should only set the configurations for unclean leader election if you want to leave it on forever. (i.e. you genuinely don't care about losing data, and just always want a leader.) best, Colin On Wed, Aug 28, 2024, at 23:11, Dima Brodsky wrote: > Thank you! This helps alot > > Sent from Mobile > > > On Wed, Aug 28, 2024 at 19:48 Luke Chen <show...@gmail.com> wrote: > >> Hi Dima, >> >> Before v3.9.0, brokers in KRaft mode didn't honor >> `unclean.leader.election.enable` config. >> So, the static config and dynamic config didn't work. >> The only thing works before v3.9.0 is using he admin API or cli to run >> `kafka-leader-election.sh` with `unclean` option. >> >> Thanks. >> Luke >> >> On Thu, Aug 29, 2024 at 10:39 AM Dima Brodsky <ddbrod...@gmail.com> wrote: >> >> > Thanks Luke, >> > >> > I meant in pre 3.9.x, like 3.6.x or 3.7.x branch for example. >> > >> > ttyl >> > Dima >> > >> > >> > On Wed, Aug 28, 2024 at 7:36 PM Luke Chen <show...@gmail.com> wrote: >> > >> > > Hi Dima, >> > > >> > > > I assume that unclean leader election exists as a static config in >> > kafka >> > > kraft, but is still unavailable dynamically? Or is it unavailable in >> > > general? >> > > >> > > Let me make it clear. In v3.9.0, the `unclean.leader.election.enable` >> in >> > > KRaft will work, either static config or dynamic config. >> > > It's just the behavior changes when dynamically enabling the >> > > `unclean.leader.election.enable`, which will not trigger unclean leader >> > > election immediately after config change. The detailed behavior change >> is >> > > as described above. >> > > >> > > Let me know if it's still not clear. >> > > >> > > Thanks. >> > > Luke >> > > >> > > >> > > On Thu, Aug 29, 2024 at 1:58 AM Dima Brodsky <ddbrod...@gmail.com> >> > wrote: >> > > >> > > > Hi Luke, >> > > > >> > > > I assume that unclean leader election exists as a static config in >> > kafka >> > > > kraft, but is still unavailable dynamically? Or is it unavailable in >> > > > general? >> > > > >> > > > Thanks! >> > > > ttyl >> > > > Dima >> > > > >> > > > >> > > > On Wed, Aug 28, 2024 at 3:45 AM Luke Chen <show...@gmail.com> wrote: >> > > > >> > > > > Hi all, >> > > > > >> > > > > In v3.9.0, we're planning to support the >> > > `unclean.leader.election.enable` >> > > > > config in KRaft. Currently, the implementation PR >> > > > > <https://github.com/apache/kafka/pull/16866> is under review now. >> > But >> > > > I'd >> > > > > like to raise the discussion to let you know there will be one >> > behavior >> > > > > change when enabling `unclean.leader.election.enable` config >> > > dynamically >> > > > in >> > > > > KRaft. >> > > > > >> > > > > scenario: >> > > > > 1. broker [0, 1] config are all set >> > > > `unclean.leader.election.enable=false`. >> > > > > 2. topic A is created with 1 partition, 2 replication factors and >> > > > > `unclean.leader.election.enable=false`. Leader of the partition 0 >> is >> > 0. >> > > > ISR >> > > > > is [0]. >> > > > > 3. broker 0 is down, which leaves ISR to empty [ ], and leader >> none. >> > > > > 4. dynamically alter config using admin API to set >> > > > > `unclean.leader.election.enable=true`, no matter it changes to the >> > > topic >> > > > A, >> > > > > or to broker 1, or default broker. >> > > > > >> > > > > In ZK mode, with this scenario, it'll trigger unclean leader >> election >> > > > > immediately, and broker 1 will be elected as the leader, which >> brings >> > > the >> > > > > availability of the partition back. >> > > > > In KRaft mode, with this scenario, the unclean leader election will >> > be >> > > > > triggered in an unclean election thread, which is scheduled every 5 >> > > mins >> > > > by >> > > > > default. That is, this partition might still be unavailable even if >> > it >> > > > > enables `unclean.leader.election.enable`. If users want to elect a >> > > leader >> > > > > for this partition, run `kafka-leader-election.sh` with `unclean` >> > > option >> > > > to >> > > > > trigger the unclean leader election immediately. >> > > > > >> > > > > The main reason we didn't implement what ZK did is because of this >> > > > > complication >> > > > > <https://github.com/apache/kafka/pull/16284#discussion_r1635410206 >> > >> > we >> > > > > faced. >> > > > > >> > > > > Other behaviors keep the same. Ex: When the topic config, or broker >> > > > config, >> > > > > or default broker config set `unclean.leader.election.enable=true`, >> > and >> > > > > later, a partition has leader down with empty ISR [ ], and having >> > some >> > > > > alive non-ISR replicas, in this case, the unclean leader election >> > will >> > > be >> > > > > triggered, as long as the config was set earlier. >> > > > > >> > > > > But since the PR is still under review, any comments or thoughts >> are >> > > > > welcomed. >> > > > > >> > > > > Thank you. >> > > > > Luke >> > > > > >> > > > >> > > > >> > > > -- >> > > > ddbrod...@gmail.com >> > > > >> > > > "The price of reliability is the pursuit of the utmost simplicity. >> > > > It is a price which the very rich find the most hard to pay." >> > > > >> (Sir >> > > > Antony Hoare, 1980) >> > > > >> > > >> > >> > >> > -- >> > ddbrod...@gmail.com >> > >> > "The price of reliability is the pursuit of the utmost simplicity. >> > It is a price which the very rich find the most hard to pay." >> > (Sir >> > Antony Hoare, 1980) >> > >>
