Hi Sophie and David, Thank you both for your replies,
You are in the best position to know if this can be useful or not ;) and I completely agree with your comments, it seems to be too specific to our use cases. Therefore, I propose to end the discussion and see if in the future any other people have a similar use case. I will also change the status of the KIP to Rejected, close the related PR and Jira issue . Have a nice day, Best regards, Mathieu Le mar. 25 oct. 2022 à 23:19, Sophie Blee-Goldman <sop...@confluent.io.invalid> a écrit : > Thanks for clarifying Mathieu, I hadn't seen your PR yet and assumed from > your > earlier comments that this was still a WIP. > > I do tend to agree with David on this though, I was also struggling to > understand the > use case/motivation and felt it was quite specific. As I think David said > in his PR > comment, we've tried to limit the assignors we include to more common and > simple > use cases that are easy for people to understand and choose between. > > Of course since there is now a PR for it, if any others do have a similar > use case and > would find it helpful, they can plug in a custom assignor based on your > patch. If we do > see this happening often enough that people start requesting it be added to > the OOTB > assignors, then it would make sense to revisit this PR. As it is, I suspect > just putting it > out there for others to use via the pluggable interface may be helpful > enough. > > Either way, thanks for the contribution! > > -Sophie > > On Mon, Oct 24, 2022 at 11:19 PM David Jacot <david.ja...@gmail.com> > wrote: > > > Hi Mathieu, > > > > Thanks for the effort that you have put in creating this KIP. I just read > > it again and I am still confused by the use cases and the motivation. I > > suppose that this works for your data model but it does not seem to be a > > general pattern. > > > > Overall, I stick to the comment that I made in the PR. I still feel like > > that this is something too use case specific to make it into Apache > Kafka. > > > > Best, > > David > > > > Le mar. 25 oct. 2022 à 07:25, Mathieu Amblard <mathieu.ambl...@gmail.com > > > > a > > écrit : > > > > > Hi Sophie, > > > > > > Thank you for your message. > > > > > > The TopicRoundRobinAssignor has been already implemented (because the > > > existing ones do not address our use cases correctly), it implements > the > > > interface you mentionned, and I have made a PR to the Kafka repo where > > you > > > can see the source code. It was in a comment of that PR that someone > from > > > the core team proposes to create a KIP. > > > > > > I have also fully unit tested this assignor (see the PR). Moreover, > > around > > > 20 microservices are using it in production, for 6 months now, in the > > > company I am working for. So I think it has been proven and that's why > I > > > have made this proposal. I would never have dared to suggest something > I > > > have never tested and if there is no need for it. > > > > > > Finally, this assignor suits well our Kafka architecture and use cases > > > contrary to the existing ones. For reminder, we are using Kafka as a > > > messaging system (no streaming) and we can have only one type of > message > > > per topic. Maybe it is too specific, maybe not, that's also why I have > > made > > > this KIP, to challenge it, to see if someone has the same needs and if > > this > > > assignor can help others. If not, that's not a problem, I will simply > > keep > > > it in our source code. > > > > > > Regards, > > > Mathieu > > > > > > > > > Le mar. 25 oct. 2022 à 01:51, Sophie Blee-Goldman > > > <sop...@confluent.io.invalid> a écrit : > > > > > > > Hey Mathieu, > > > > > > > > Apologies if you already know this, but the partition assignor > > interface > > > is > > > > fully pluggable. > > > > That means you can plug in a custom PartitionAssignor implementation, > > > > without > > > > having to go through a KIP or commit any code to the AK repo. > > > > > > > > I suspect some of the current unknowns and solutions will become > clear > > > when > > > > you start > > > > to actually write this assignor and test it out in your environment. > > Then > > > > you can play around > > > > with what works and what doesn't work, and come back to the KIP if > > > desired > > > > with a stronger > > > > argument for why it's needed. Or you can just take your assignor and > > > > publish it in a public > > > > git repo for anyone who might have a similar use case as you. > > > > > > > > Just my two cents, I'd recommend in this case you start with the > > > > implementation before > > > > worrying about donating your assignor back to the main repo with a > KIP. > > > IF > > > > you do want to, > > > > it would then be much easier to convince people when they can see > your > > > > assignor logic > > > > for themselves, and you'll be able to answer any questions. > > > > > > > > Best, > > > > Sophie > > > > > > > > On Fri, Oct 21, 2022 at 2:21 AM Mathieu Amblard < > > > mathieu.ambl...@gmail.com > > > > > > > > > wrote: > > > > > > > > > Hello everybody, > > > > > > > > > > Just to let you know that I have added a chapter about having > > multiple > > > > > containers (multiple pods for Kubernetes) running the same > > application > > > : > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-874%3A+TopicRoundRobinAssignor#KIP874:TopicRoundRobinAssignor-Howdoesitworkifwehavemultiplecontainersrunningthesameapplication > > > > > ? > > > > > > > > > > Regards, > > > > > Mathieu > > > > > > > > > > Le mar. 11 oct. 2022 à 20:00, Mathieu Amblard < > > > mathieu.ambl...@gmail.com > > > > > > > > > > a > > > > > écrit : > > > > > > > > > > > Hi Hector, > > > > > > > > > > > > > > > > > > > > > > > > First, thank you for your questions ! > > > > > > > > > > > > > > > > > > > > > > > > *If the goal is to do the partition assignments at a topic level, > > > > > wouldn't > > > > > > having single-partition topics solve this problem?* > > > > > > > > > > > > > > > > > > > > > > > > We are in a microservices environment; therefore, we can have > > > multiple > > > > > > containers running the same application. > > > > > > > > > > > > Using the TopicRoundRobinAssignor, the partitions are uniformly > > > > balanced > > > > > > to each container, and we can get better performances. > > > > > > > > > > > > Let suppose there are 2 instances of the same application A0 and > > A1, > > > 2 > > > > > > consumers C0 and C1, and two topics t0 and t1. t0 has 3 > partitions > > > and > > > > t1 > > > > > > has two partitions resulting in partitions : t0p0, t0p1, t0p2, > > t1p0, > > > > > t1p1. > > > > > > > > > > > > If we use the TopicRoundRobinAssignor, the assignment will be : > > > > > > > > > > > > A0 : [ C0: [t0p0, t0p2], C1: [t1p0] ] > > > > > > > > > > > > A1 : [ C0: [t0p1], C1: [t1p1] ] > > > > > > > > > > > > > > > > > > > > > > > > *How will the group leader know that T2 should not be re-assigned > > on > > > > the > > > > > > next rebalance? Can you elaborate a bit more on the mechanisms > used > > > to > > > > > > communicate this state to the other group members?* > > > > > > > > > > > > > > > > > > > > > > > > Currently, the group leader will not know that T2 should not be > > > > > > re-assigned on the next balance. > > > > > > > > > > > > For this first iteration, we simply keep in memory that T2 has a > > > poison > > > > > > pill and therefore we ignore all incoming messages from T2. We > > > > basically > > > > > > consume them without acknowledging them. > > > > > > > > > > > > As you can imagine, in the case of having multiple instances of > the > > > > same > > > > > > application, in case of error, the partition will be rebalanced > to > > > > > another > > > > > > instance. > > > > > > > > > > > > Nevertheless, this is not really a problem (at least for our use > > > > cases), > > > > > > as soon as the poison pill is consumed, the consumer of this > other > > > > > instance > > > > > > will be stopped, and so on, and so on. It will take a few tens of > > > > seconds > > > > > > before the last consumer of the poison pill will be stopped and > so > > > the > > > > > > consumption of the entire topic. > > > > > > > > > > > > For a second iteration, we have planned to find a solution to > avoid > > > > this > > > > > > time lapse between the consumer of the first instance being > stopped > > > and > > > > > the > > > > > > last one. Currently, I do not have a solution, but we are > thinking > > > > about > > > > > > different options, avoiding rebalancing partitions containing a > > > poison > > > > > pill > > > > > > is one of them. > > > > > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > > > Mathieu > > > > > > > > > > > > Le ven. 7 oct. 2022 à 16:54, Hector Geraldino (BLOOMBERG/ 919 3RD > > A) > > > < > > > > > > hgerald...@bloomberg.net> a écrit : > > > > > > > > > > > >> Hi Mathieu. I took a look at your KIP and have a couple > questions. > > > > > >> > > > > > >> If the goal is to do the partition assignments at a topic level, > > > > > wouldn't > > > > > >> having single-partition topics solve this problem? > > > > > >> > > > > > >> You also mentioned that your goal is to minimize the potential > of > > a > > > > > >> poison pill message breaking all members of a group (by keeping > > > track > > > > of > > > > > >> which topics have 'failed'), but it is not clear how this can be > > > > > achieved > > > > > >> with this assignor. If we imagine an scenario where: > > > > > >> > > > > > >> * A group has 3 members (A, B, C) > > > > > >> * Members are subscribed to 3 topics (T1, T2, T3) > > > > > >> * Each member is assigned one topic (A[T1], B[T2], C[T3]) > > > > > >> * One member fails to consume from a topic/partition (B[T2]), > and > > > goes > > > > > >> into failed state > > > > > >> > > > > > >> How will the group leader know that T2 should not be re-assigned > > on > > > > the > > > > > >> next rebalance? Can you elaborate a bit more on the mechanisms > > used > > > to > > > > > >> communicate this state to the other group members? > > > > > >> > > > > > >> Thanks > > > > > >> > > > > > >> From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: > > > > > >> dev@kafka.apache.org > > > > > >> Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor > > > > > >> > > > > > >> Hi Kafka Developers, > > > > > >> > > > > > >> My proposal is to add a new partition assignment strategy at the > > > topic > > > > > >> level to : > > > > > >> - have a better data consistency by consumed topic in case of > > > > exception > > > > > >> - have a solution much thread safe for the consumer > > > > > >> In case there are multiple consumers and multiple topics. > > > > > >> > > > > > >> Here is the link to the KIP with all the explanations : > > > > > >> https://cwiki.apache.org/confluence/x/XozGDQ > > > > > >> > > > > > >> Thank you in advance for your feedbacks, > > > > > >> Mathieu > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
