Hi Sagar,
I have some thoughts about Kafka Connect integrating with KIP-848, but I
think we should have a separate discussion thread for the Kafka Connect
KIP: Integrating Kafka Connect With New Consumer Rebalance Protocol [1],
and let this discussion thread focus on consumer rebalance protocol, WDYT?
[1]
https://cwiki.apache.org/confluence/display/KAFKA/%5BDRAFT%5DIntegrating+Kafka+Connect+With+New+Consumer+Rebalance+Protocol
Thank you.
Luke
On Fri, Aug 12, 2022 at 9:31 PM Sagar wrote:
> Thank you Guozhang/David for the feedback. Looks like there's agreement on
> using separate APIs for Connect. I would revisit the doc and see what
> changes are to be made.
>
> Thanks!
> Sagar.
>
> On Tue, Aug 9, 2022 at 7:11 PM David Jacot
> wrote:
>
> > Hi Sagar,
> >
> > Thanks for the feedback and the document. That's really helpful. I
> > will take a look at it.
> >
> > Overall, it seems to me that both Connect and the Consumer could share
> > the same underlying "engine". The main difference is that the Consumer
> > assigns topic-partitions to members whereas Connect assigns tasks to
> > workers. I see two ways to move forward:
> > 1) We extend the new proposed APIs to support different resource types
> > (e.g. partitions, tasks, etc.); or
> > 2) We use new dedicated APIs for Connect. The dedicated APIs would be
> > similar to the new ones but different on the content/resources and
> > they would rely on the same engine on the coordinator side.
> >
> > I personally lean towards 2) because I am not a fan of overcharging
> > APIs to serve different purposes. That being said, I am not opposed to
> > 1) if we can find an elegant way to do it.
> >
> > I think that we can continue to discuss it here for now in order to
> > ensure that this KIP is compatible with what we will do for Connect in
> > the future.
> >
> > Best,
> > David
> >
> > On Mon, Aug 8, 2022 at 2:41 PM David Jacot wrote:
> > >
> > > Hi all,
> > >
> > > I am back from vacation. I will go through and address your comments
> > > in the coming days. Thanks for your feedback.
> > >
> > > Cheers,
> > > David
> > >
> > > On Wed, Aug 3, 2022 at 10:05 PM Gregory Harris
> > wrote:
> > > >
> > > > Hey All!
> > > >
> > > > Thanks for the KIP, it's wonderful to see cooperative rebalancing
> > making it
> > > > down the stack!
> > > >
> > > > I had a few questions:
> > > >
> > > > 1. The 'Rejected Alternatives' section describes how member epoch
> > should
> > > > advance in step with the group epoch and assignment epoch values. I
> > think
> > > > that this is a good idea for the reasons described in the KIP. When
> the
> > > > protocol is incrementally assigning partitions to a worker, what
> member
> > > > epoch does each incremental assignment use? Are member epochs
> re-used,
> > and
> > > > a single member epoch can correspond to multiple different
> > (monotonically
> > > > larger) assignments?
> > > >
> > > > 2. Is the Assignor's 'Reason' field opaque to the group coordinator?
> If
> > > > not, should custom client-side assignor implementations interact with
> > the
> > > > Reason field, and how is its common meaning agreed upon? If so, what
> > is the
> > > > benefit of a distinct Reason field over including such functionality
> > in the
> > > > opaque metadata?
> > > >
> > > > 3. The following is included in the KIP: "Thanks to this, the input
> of
> > the
> > > > client side assignor is entirely driven by the group coordinator. The
> > > > consumer is no longer responsible for maintaining any state besides
> its
> > > > assigned partitions." Does this mean that the client-side assignor
> MAY
> > > > incorporate additional non-Metadata state (such as partition
> > throughput,
> > > > cpu/memory metrics, config topics, etc), or that additional
> > non-Metadata
> > > > state SHOULD NOT be used?
> > > >
> > > > 4. I see that there are separate classes
> > > > for org.apache.kafka.server.group.consumer.PartitionAssignor
> > > > and org.apache.kafka.clients.consumer.PartitionAssignor that seem to
> > > > overlap significantly. Is it possible for these two implementations
> to
> > be
> > > > unified? This would serve to promote feature parity of server-side
> and
> > > > client-side assignors, and would also facilitate operational
> > flexibility in
> > > > certain situations. For example, if a server-side assignor has some
> > poor
> > > > behavior and needs a patch, deploying the patched assignor to the
> > client
> > > > and switching one consumer group to a client-side assignor may be
> > faster
> > > > and less risky than patching all of the brokers. With the currently
> > > > proposed distinct APIs, a non-trivial reimplementation would have to
> be
> > > > assembled, and if the two APIs have diverged significantly, then it
> is
> > > > possible that a reimplementation would not be possible.
> > > >
> > > > --
> > > > Greg Harris
> > > > gharris1...@gmail.com
> > > > github.com/gharris1727
> > > >
> > > > On Wed, Aug 3, 2022 at 8:39 AM Sagar
> > wr
