Hi Sophie, Thank you for your message.
The TopicRoundRobinAssignor has been already implemented (because the existing ones do not address our use cases correctly), it implements the interface you mentionned, and I have made a PR to the Kafka repo where you can see the source code. It was in a comment of that PR that someone from the core team proposes to create a KIP. I have also fully unit tested this assignor (see the PR). Moreover, around 20 microservices are using it in production, for 6 months now, in the company I am working for. So I think it has been proven and that's why I have made this proposal. I would never have dared to suggest something I have never tested and if there is no need for it. Finally, this assignor suits well our Kafka architecture and use cases contrary to the existing ones. For reminder, we are using Kafka as a messaging system (no streaming) and we can have only one type of message per topic. Maybe it is too specific, maybe not, that's also why I have made this KIP, to challenge it, to see if someone has the same needs and if this assignor can help others. If not, that's not a problem, I will simply keep it in our source code. Regards, Mathieu Le mar. 25 oct. 2022 à 01:51, Sophie Blee-Goldman <sop...@confluent.io.invalid> a écrit : > Hey Mathieu, > > Apologies if you already know this, but the partition assignor interface is > fully pluggable. > That means you can plug in a custom PartitionAssignor implementation, > without > having to go through a KIP or commit any code to the AK repo. > > I suspect some of the current unknowns and solutions will become clear when > you start > to actually write this assignor and test it out in your environment. Then > you can play around > with what works and what doesn't work, and come back to the KIP if desired > with a stronger > argument for why it's needed. Or you can just take your assignor and > publish it in a public > git repo for anyone who might have a similar use case as you. > > Just my two cents, I'd recommend in this case you start with the > implementation before > worrying about donating your assignor back to the main repo with a KIP. IF > you do want to, > it would then be much easier to convince people when they can see your > assignor logic > for themselves, and you'll be able to answer any questions. > > Best, > Sophie > > On Fri, Oct 21, 2022 at 2:21 AM Mathieu Amblard <mathieu.ambl...@gmail.com > > > wrote: > > > Hello everybody, > > > > Just to let you know that I have added a chapter about having multiple > > containers (multiple pods for Kubernetes) running the same application : > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-874%3A+TopicRoundRobinAssignor#KIP874:TopicRoundRobinAssignor-Howdoesitworkifwehavemultiplecontainersrunningthesameapplication > > ? > > > > Regards, > > Mathieu > > > > Le mar. 11 oct. 2022 à 20:00, Mathieu Amblard <mathieu.ambl...@gmail.com > > > > a > > écrit : > > > > > Hi Hector, > > > > > > > > > > > > First, thank you for your questions ! > > > > > > > > > > > > *If the goal is to do the partition assignments at a topic level, > > wouldn't > > > having single-partition topics solve this problem?* > > > > > > > > > > > > We are in a microservices environment; therefore, we can have multiple > > > containers running the same application. > > > > > > Using the TopicRoundRobinAssignor, the partitions are uniformly > balanced > > > to each container, and we can get better performances. > > > > > > Let suppose there are 2 instances of the same application A0 and A1, 2 > > > consumers C0 and C1, and two topics t0 and t1. t0 has 3 partitions and > t1 > > > has two partitions resulting in partitions : t0p0, t0p1, t0p2, t1p0, > > t1p1. > > > > > > If we use the TopicRoundRobinAssignor, the assignment will be : > > > > > > A0 : [ C0: [t0p0, t0p2], C1: [t1p0] ] > > > > > > A1 : [ C0: [t0p1], C1: [t1p1] ] > > > > > > > > > > > > *How will the group leader know that T2 should not be re-assigned on > the > > > next rebalance? Can you elaborate a bit more on the mechanisms used to > > > communicate this state to the other group members?* > > > > > > > > > > > > Currently, the group leader will not know that T2 should not be > > > re-assigned on the next balance. > > > > > > For this first iteration, we simply keep in memory that T2 has a poison > > > pill and therefore we ignore all incoming messages from T2. We > basically > > > consume them without acknowledging them. > > > > > > As you can imagine, in the case of having multiple instances of the > same > > > application, in case of error, the partition will be rebalanced to > > another > > > instance. > > > > > > Nevertheless, this is not really a problem (at least for our use > cases), > > > as soon as the poison pill is consumed, the consumer of this other > > instance > > > will be stopped, and so on, and so on. It will take a few tens of > seconds > > > before the last consumer of the poison pill will be stopped and so the > > > consumption of the entire topic. > > > > > > For a second iteration, we have planned to find a solution to avoid > this > > > time lapse between the consumer of the first instance being stopped and > > the > > > last one. Currently, I do not have a solution, but we are thinking > about > > > different options, avoiding rebalancing partitions containing a poison > > pill > > > is one of them. > > > > > > > > > > > > Cheers, > > > > > > Mathieu > > > > > > Le ven. 7 oct. 2022 à 16:54, Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > hgerald...@bloomberg.net> a écrit : > > > > > >> Hi Mathieu. I took a look at your KIP and have a couple questions. > > >> > > >> If the goal is to do the partition assignments at a topic level, > > wouldn't > > >> having single-partition topics solve this problem? > > >> > > >> You also mentioned that your goal is to minimize the potential of a > > >> poison pill message breaking all members of a group (by keeping track > of > > >> which topics have 'failed'), but it is not clear how this can be > > achieved > > >> with this assignor. If we imagine an scenario where: > > >> > > >> * A group has 3 members (A, B, C) > > >> * Members are subscribed to 3 topics (T1, T2, T3) > > >> * Each member is assigned one topic (A[T1], B[T2], C[T3]) > > >> * One member fails to consume from a topic/partition (B[T2]), and goes > > >> into failed state > > >> > > >> How will the group leader know that T2 should not be re-assigned on > the > > >> next rebalance? Can you elaborate a bit more on the mechanisms used to > > >> communicate this state to the other group members? > > >> > > >> Thanks > > >> > > >> From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: > > >> dev@kafka.apache.org > > >> Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor > > >> > > >> Hi Kafka Developers, > > >> > > >> My proposal is to add a new partition assignment strategy at the topic > > >> level to : > > >> - have a better data consistency by consumed topic in case of > exception > > >> - have a solution much thread safe for the consumer > > >> In case there are multiple consumers and multiple topics. > > >> > > >> Here is the link to the KIP with all the explanations : > > >> https://cwiki.apache.org/confluence/x/XozGDQ > > >> > > >> Thank you in advance for your feedbacks, > > >> Mathieu > > >> > > >> > > >> > > >
