Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[Ismael Juma]

Three quick comments:

1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we should
document the differences in more detail before deciding one way or another.
That said, if people pass java.util.regex.Pattern, they expect their
semantics to be honored. If we are doing something different, then we
should consider adding an overload with our own Pattern class (I don't
think we'd want to expose re2j's at this point).
2. Regarding topic ids, any major new protocol should integrate fully with
it and should handle the topic recreation case correctly. That's the main
part we need to handle. I agree with David that we'd want to add topic ids
to the relevant protocols that don't have it yet and that we can probably
focus on the internals versus adding new APIs to the Java Consumer (unless
we find that adding new APIs is required for reasonable semantics).
3. I am still not sure about the coordinator storing the configs. It's
powerful for configs to be centralized in the metadata log for various
reasons (auditability, visibility, consistency, etc.). Similarly, I am not
sure about automatically deleting configs in a way that they cannot be
recovered. A good property for modern systems is to minimize the number of
unrecoverable data loss scenarios.

Ismael

On Wed, Jul 13, 2022 at 3:47 PM David Jacot 
wrote:

> Thanks Guozhang. My answers are below:
>
> > 1) the migration path, especially the last step when clients flip the
> flag
> > to enable the new protocol, in which we would have a window where both
> new
> > protocols / rpcs and old protocols / rpcs are used by members of the same
> > group. How the coordinator could "mimic" the old behavior while using the
> > new protocol is something we need to present about.
>
> Noted. I just published a new version of KIP which includes more
> details about this. See the "Supporting Online Consumer Group Upgrade"
> and the "Compatibility, Deprecation, and Migration Plan". I think that
> I have to think through a few cases now but the overall idea and
> mechanism should be understandable.
>
> > 2) the usage of topic ids. So far as KIP-516 the topic ids are only used
> as
> > part of RPCs and admin client, but they are not exposed via any public
> APIs
> > to consumers yet. I think the question is, first should we let the
> consumer
> > client to be maintaining the names -> ids mapping itself to fully
> leverage
> > on all the augmented existing RPCs and the new RPCs with the topic ids;
> and
> > secondly, should we ever consider exposing the topic ids in the consumer
> > public APIs as well (both subscribe/assign, as well as in the rebalance
> > listener for cases like topic deletion-and-recreation).
>
> a) Assuming that we would include converting all the offsets related
> RPCs to using topic ids in this KIP, the consumer would be able to
> fully operate with topic ids. That being said, it still has to provide
> the topics names in various APIs so having a mapping in the consumer
> seems inevitable to me.
> b) I don't have a strong opinion on this. Here I wonder if this goes
> beyond the scope of this KIP. I would rather focus on the internals
> here and we can consider this separately if we see value in doing it.
>
> Coming back to Ismael's point about using topic ids in the
> ConsumerGroupHeartbeatRequest, I think that there is one advantage in
> favour of it. The consumer will have the opportunity to validate that
> the topics exists before passing them into the group rebalance
> protocol. Obviously, the coordinator will also notice it but it does
> not really have a way to reject an invalid topic in the response.
>
> > I'm agreeing with David on all other minor questions except for the
> > `subscribe(Pattern)` question: personally I think it's not necessary to
> > deprecate the subscribe API with Pattern, but instead we still use
> Pattern
> > while just documenting that our subscription may be rejected by the
> server.
> > Since the incompatible case is a very rare scenario I felt using an
> > overloaded `String` based subscription may be more vulnerable to various
> > invalid regexes.
>
> That could work. I have to look at the differences between the two
> engines to better understand the potential issues. My understanding is
> that would work for all the basic regular expressions. The differences
> between the two are mainly about the various character classes. I
> wonder what other people think about this.
>
> Best,
> David
>
> On Tue, Jul 12, 2022 at 11:28 PM Guozhang Wang  wrote:
> >
> > Thanks David! I think on the high level there are two meta points we need
> > to concretize a bit more:
> >
> > 1) the migration path, especially the last step when clients flip the
> flag
> > to enable the new protocol, in which we would have a window where both
> new
> > protocols / rpcs and old protocols / rpcs are used by members of the same
> > group. How the coordinator could "mimic" the old behavior while using the
> > new protocol is something we need to pre

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

Thanks Hector! Our goal is to move forward with specialized API
instead of relying on one generic API. For Connect, we can apply the
exact same pattern and reuse/share the core implementation on the
server side. For the schema registry, I think that we should consider
having a tailored API to do simple membership/leader election.

Best,
David

On Fri, Jul 15, 2022 at 10:22 AM Ismael Juma  wrote:
>
> Three quick comments:
>
> 1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we should
> document the differences in more detail before deciding one way or another.
> That said, if people pass java.util.regex.Pattern, they expect their
> semantics to be honored. If we are doing something different, then we
> should consider adding an overload with our own Pattern class (I don't
> think we'd want to expose re2j's at this point).
> 2. Regarding topic ids, any major new protocol should integrate fully with
> it and should handle the topic recreation case correctly. That's the main
> part we need to handle. I agree with David that we'd want to add topic ids
> to the relevant protocols that don't have it yet and that we can probably
> focus on the internals versus adding new APIs to the Java Consumer (unless
> we find that adding new APIs is required for reasonable semantics).
> 3. I am still not sure about the coordinator storing the configs. It's
> powerful for configs to be centralized in the metadata log for various
> reasons (auditability, visibility, consistency, etc.). Similarly, I am not
> sure about automatically deleting configs in a way that they cannot be
> recovered. A good property for modern systems is to minimize the number of
> unrecoverable data loss scenarios.
>
> Ismael
>
> On Wed, Jul 13, 2022 at 3:47 PM David Jacot 
> wrote:
>
> > Thanks Guozhang. My answers are below:
> >
> > > 1) the migration path, especially the last step when clients flip the
> > flag
> > > to enable the new protocol, in which we would have a window where both
> > new
> > > protocols / rpcs and old protocols / rpcs are used by members of the same
> > > group. How the coordinator could "mimic" the old behavior while using the
> > > new protocol is something we need to present about.
> >
> > Noted. I just published a new version of KIP which includes more
> > details about this. See the "Supporting Online Consumer Group Upgrade"
> > and the "Compatibility, Deprecation, and Migration Plan". I think that
> > I have to think through a few cases now but the overall idea and
> > mechanism should be understandable.
> >
> > > 2) the usage of topic ids. So far as KIP-516 the topic ids are only used
> > as
> > > part of RPCs and admin client, but they are not exposed via any public
> > APIs
> > > to consumers yet. I think the question is, first should we let the
> > consumer
> > > client to be maintaining the names -> ids mapping itself to fully
> > leverage
> > > on all the augmented existing RPCs and the new RPCs with the topic ids;
> > and
> > > secondly, should we ever consider exposing the topic ids in the consumer
> > > public APIs as well (both subscribe/assign, as well as in the rebalance
> > > listener for cases like topic deletion-and-recreation).
> >
> > a) Assuming that we would include converting all the offsets related
> > RPCs to using topic ids in this KIP, the consumer would be able to
> > fully operate with topic ids. That being said, it still has to provide
> > the topics names in various APIs so having a mapping in the consumer
> > seems inevitable to me.
> > b) I don't have a strong opinion on this. Here I wonder if this goes
> > beyond the scope of this KIP. I would rather focus on the internals
> > here and we can consider this separately if we see value in doing it.
> >
> > Coming back to Ismael's point about using topic ids in the
> > ConsumerGroupHeartbeatRequest, I think that there is one advantage in
> > favour of it. The consumer will have the opportunity to validate that
> > the topics exists before passing them into the group rebalance
> > protocol. Obviously, the coordinator will also notice it but it does
> > not really have a way to reject an invalid topic in the response.
> >
> > > I'm agreeing with David on all other minor questions except for the
> > > `subscribe(Pattern)` question: personally I think it's not necessary to
> > > deprecate the subscribe API with Pattern, but instead we still use
> > Pattern
> > > while just documenting that our subscription may be rejected by the
> > server.
> > > Since the incompatible case is a very rare scenario I felt using an
> > > overloaded `String` based subscription may be more vulnerable to various
> > > invalid regexes.
> >
> > That could work. I have to look at the differences between the two
> > engines to better understand the potential issues. My understanding is
> > that would work for all the basic regular expressions. The differences
> > between the two are mainly about the various character classes. I
> > wonder what other people th

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1072

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 568210 lines...]
[2022-07-15T10:37:14.564Z] 
[2022-07-15T10:37:14.564Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_true] PASSED
[2022-07-15T10:37:14.564Z] 
[2022-07-15T10:37:14.564Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_true] STARTED
[2022-07-15T10:37:44.457Z] 
[2022-07-15T10:37:44.457Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_UPDATE_false] PASSED
[2022-07-15T10:37:44.457Z] 
[2022-07-15T10:37:44.457Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_CLOSE_true] STARTED
[2022-07-15T10:37:44.457Z] 
[2022-07-15T10:37:44.457Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_CLOSE_true] PASSED
[2022-07-15T10:37:44.457Z] 
[2022-07-15T10:37:44.457Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_CLOSE_true] STARTED
[2022-07-15T10:37:44.457Z] 
[2022-07-15T10:37:44.457Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_CLOSE_true] PASSED
[2022-07-15T10:37:44.457Z] 
[2022-07-15T10:37:44.457Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_true] STARTED
[2022-07-15T10:37:45.482Z] 
[2022-07-15T10:37:45.482Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_true] PASSED
[2022-07-15T10:37:45.482Z] 
[2022-07-15T10:37:45.483Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_true] STARTED
[2022-07-15T10:38:11.687Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_true] PASSED
[2022-07-15T10:38:11.688Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_CLOSE_false] STARTED
[2022-07-15T10:38:11.688Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldThrowUnlimitedWindows[ON_WINDOW_CLOSE_false] PASSED
[2022-07-15T10:38:11.688Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_CLOSE_false] STARTED
[2022-07-15T10:38:11.688Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithNoGrace[ON_WINDOW_CLOSE_false] PASSED
[2022-07-15T10:38:11.688Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_false] STARTED
[2022-07-15T10:38:11.688Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldAggregateWindowedWithGrace[ON_WINDOW_CLOSE_false] PASSED
[2022-07-15T10:38:11.688Z] 
[2022-07-15T10:38:11.688Z] 
org.apache.kafka.streams.integration.TimeWindowedKStreamIntegrationTest > 
shouldRestoreAfterJoinRestart[ON_WINDOW_CLOSE_false] STARTED
[2022-07-15T10:38:30.784Z] 
[2022-07-15T10:38:30.784Z] 
org.apache.kafka.streams.integration.StreamsUpgradeTestIntegrationTest > 
testVersionProbingUpgrade PASSED
[2022-07-15T10:38:30.784Z] 
[2022-07-15T10:38:30.784Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes STARTED
[2022-07-15T10:38:30.784Z] 
[2022-07-15T10:38:30.784Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldInheritSerdes PASSED
[2022-07-15T10:38:30.784Z] 
[2022-07-15T10:38:30.784Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated STARTED
[2022-07-15T10:38:34.385Z] 
[2022-07-15T10:38:34.385Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldShutdownWhenRecordConstraintIsViolated PASSED
[2022-07-15T10:38:34.385Z] 
[2022-07-15T10:38:34.385Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes STARTED
[2022-07-15T10:38:37.986Z] 
[2022-07-15T10:38:37.986Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldUseDefaultSerdes PASSED
[2022-07-15T10:38:37.986Z] 
[2022-07-15T10:38:37.986Z] 
org.apache.kafka.streams.integration.SuppressionIntegrationTest > 
shouldAllowDisablingChangelog STARTED
[2022-07-15T10:38:38.457Z] 
[2022-07-15T10:38:38.457Z] 
org.apache.kafka.stre

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #8

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 567796 lines...]
‘builds30’ doesn’t have label ‘builds26’
‘builds32’ doesn’t have label ‘builds26’
‘builds33’ doesn’t have label ‘builds26’
‘builds34’ doesn’t have label ‘builds26’
‘builds36’ doesn’t have label ‘builds26’
‘builds37’ doesn’t have label ‘builds26’
‘builds38’ doesn’t have label ‘builds26’
‘builds39’ doesn’t have label ‘builds26’
‘builds40’ doesn’t have label ‘builds26’
‘builds41’ doesn’t have label ‘builds26’
‘builds42’ doesn’t have label ‘builds26’
‘builds43’ doesn’t have label ‘builds26’
‘builds44’ doesn’t have label ‘builds26’
‘builds45’ doesn’t have label ‘builds26’
‘builds46’ doesn’t have label ‘builds26’
‘builds47’ doesn’t have label ‘builds26’
‘builds48’ doesn’t have label ‘builds26’
‘builds50’ doesn’t have label ‘builds26’
‘mesos-arm1’ doesn’t have label ‘builds26’
‘openwhisk1’ doesn’t have label ‘builds26’
‘plc4x2’ doesn’t have label ‘builds26’
‘ppc64le-1’ doesn’t have label ‘builds26’
‘ubuntu-ppc64le’ doesn’t have label ‘builds26’
Waiting to resume part of Kafka » Kafka Branch Builder » 3.3 #8: Finished 
waiting
Waiting to resume part of Kafka » Kafka Branch Builder » 3.3 #8: Executor slot 
already in use
‘arm4’ doesn’t have label ‘builds26’
‘builds22’ doesn’t have label ‘builds26’
‘builds23’ doesn’t have label ‘builds26’
‘builds24’ doesn’t have label ‘builds26’
‘builds25’ doesn’t have label ‘builds26’
‘builds27’ doesn’t have label ‘builds26’
‘builds28’ doesn’t have label ‘builds26’
‘builds29’ doesn’t have label ‘builds26’
‘builds30’ doesn’t have label ‘builds26’
‘builds32’ doesn’t have label ‘builds26’
‘builds33’ doesn’t have label ‘builds26’
‘builds34’ doesn’t have label ‘builds26’
‘builds36’ doesn’t have label ‘builds26’
‘builds37’ doesn’t have label ‘builds26’
‘builds38’ doesn’t have label ‘builds26’
‘builds39’ doesn’t have label ‘builds26’
‘builds40’ doesn’t have label ‘builds26’
‘builds41’ doesn’t have label ‘builds26’
‘builds42’ doesn’t have label ‘builds26’
‘builds43’ doesn’t have label ‘builds26’
‘builds44’ doesn’t have label ‘builds26’
‘builds45’ doesn’t have label ‘builds26’
‘builds46’ doesn’t have label ‘builds26’
‘builds47’ doesn’t have label ‘builds26’
‘builds48’ doesn’t have label ‘builds26’
‘builds50’ doesn’t have label ‘builds26’
‘mesos-arm1’ doesn’t have label ‘builds26’
‘openwhisk1’ doesn’t have label ‘builds26’
‘plc4x2’ doesn’t have label ‘builds26’
‘ppc64le-1’ doesn’t have label ‘builds26’
‘ubuntu-ppc64le’ doesn’t have label ‘builds26’
Waiting to resume part of Kafka » Kafka Branch Builder » 3.3 #8: Finished 
waiting
Waiting to resume part of Kafka » Kafka Branch Builder » 3.3 #8: Executor slot 
already in use
‘arm4’ doesn’t have label ‘builds26’
‘builds22’ doesn’t have label ‘builds26’
‘builds23’ doesn’t have label ‘builds26’
‘builds24’ doesn’t have label ‘builds26’
‘builds25’ doesn’t have label ‘builds26’
‘builds27’ doesn’t have label ‘builds26’
‘builds28’ doesn’t have label ‘builds26’
‘builds29’ doesn’t have label ‘builds26’
‘builds30’ doesn’t have label ‘builds26’
‘builds32’ doesn’t have label ‘builds26’
‘builds33’ doesn’t have label ‘builds26’
‘builds34’ doesn’t have label ‘builds26’
‘builds36’ doesn’t have label ‘builds26’
‘builds37’ doesn’t have label ‘builds26’
‘builds38’ doesn’t have label ‘builds26’
‘builds39’ doesn’t have label ‘builds26’
‘builds40’ doesn’t have label ‘builds26’
‘builds41’ doesn’t have label ‘builds26’
‘builds42’ doesn’t have label ‘builds26’
‘builds43’ doesn’t have label ‘builds26’
‘builds44’ doesn’t have label ‘builds26’
‘builds45’ doesn’t have label ‘builds26’
‘builds46’ doesn’t have label ‘builds26’
‘builds47’ doesn’t have label ‘builds26’
‘builds48’ doesn’t have label ‘builds26’
‘builds50’ doesn’t have label ‘builds26’
‘mesos-arm1’ doesn’t have label ‘builds26’
‘openwhisk1’ doesn’t have label ‘builds26’
‘plc4x2’ doesn’t have label ‘builds26’
‘ppc64le-1’ doesn’t have label ‘builds26’
‘ubuntu-ppc64le’ doesn’t have label ‘builds26’
Waiting to resume part of Kafka » Kafka Branch Builder » 3.3 #8: Finished 
waiting
Waiting to resume part of Kafka » Kafka Branch Builder » 3.3 #8: Executor slot 
already in use
‘arm4’ doesn’t have label ‘builds26’
‘builds22’ doesn’t have label ‘builds26’
‘builds23’ doesn’t have label ‘builds26’
‘builds24’ doesn’t have label ‘builds26’
‘builds25’ doesn’t have label ‘builds26’
‘builds27’ doesn’t have label ‘builds26’
‘builds28’ doesn’t have label ‘builds26’
‘builds29’ doesn’t have label ‘builds26’
‘builds30’ doesn’t have label ‘builds26’
‘builds32’ doesn’t have label ‘builds26’
‘builds33’ doesn’t have label ‘builds26’
‘builds34’ doesn’t have label ‘builds26’
‘builds36’ doesn’t have label ‘builds26’
‘builds37’ doesn’t have label ‘builds26’
‘builds38’ doesn’t have label ‘builds26’
‘builds39’ doesn’t have label ‘builds26’
‘builds40’ doesn’t have label ‘builds26’
‘builds41’ doesn’t have label ‘builds26’
‘builds42’ doesn’t have label ‘builds26’
‘builds43’ doesn’

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[Sagar]

Hi David,

Thanks for the KIP. I just had minor observations:

1) In the Assignment Error section in Client Side mode Assignment process,
you mentioned => `In this case, the client side assignor can return an
error to the group coordinator`. In this case are you referring to the
Assignor returning an AssignmentError that's listed down towards the end?
If yes, do you think it would make sense to mention this explicitly here?

2) In the Case Studies section, I have a slight confusion, not sure if
others have the same. Consider this step:

When B heartbeats, the group coordinator transitions him to epoch 3 because
B has no partitions to revoke. It persists the change and reply.

   - Group (epoch=3)
  - A
  - B
  - C
   - Target Assignment (epoch=3)
  - A - partitions=[foo-0]
  - B - partitions=[foo-2]
  - C - partitions=[foo-1]
   - Member Assignment
  - A - epoch=2, partitions=[foo-0, foo-1]
  - B - epoch=3, partitions=[foo-2]
  - C - epoch=3, partitions=[foo-1]

When C heartbeats, it transitions to epoch 3 but cannot get foo-1 yet.

Here,it's mentioned that member C can't get the foo-1 partition yet, but
based on the description above, it seems it already has it. Do you think it
would be better to remove it and populate it only when it actually gets it?
I see this in a lot of other places, so have I understood it incorrectly ?


Regarding connect , it might be out of scope of this discussion, but from
what I understood it would probably be running in client side assignor mode
even on the new rebalance protocol as it has its own Custom Assignors(Eager
and IncrementalCooperative).

Thanks!

Sagar.






On Fri, Jul 15, 2022 at 5:00 PM David Jacot 
wrote:

> Thanks Hector! Our goal is to move forward with specialized API
> instead of relying on one generic API. For Connect, we can apply the
> exact same pattern and reuse/share the core implementation on the
> server side. For the schema registry, I think that we should consider
> having a tailored API to do simple membership/leader election.
>
> Best,
> David
>
> On Fri, Jul 15, 2022 at 10:22 AM Ismael Juma  wrote:
> >
> > Three quick comments:
> >
> > 1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we
> should
> > document the differences in more detail before deciding one way or
> another.
> > That said, if people pass java.util.regex.Pattern, they expect their
> > semantics to be honored. If we are doing something different, then we
> > should consider adding an overload with our own Pattern class (I don't
> > think we'd want to expose re2j's at this point).
> > 2. Regarding topic ids, any major new protocol should integrate fully
> with
> > it and should handle the topic recreation case correctly. That's the main
> > part we need to handle. I agree with David that we'd want to add topic
> ids
> > to the relevant protocols that don't have it yet and that we can probably
> > focus on the internals versus adding new APIs to the Java Consumer
> (unless
> > we find that adding new APIs is required for reasonable semantics).
> > 3. I am still not sure about the coordinator storing the configs. It's
> > powerful for configs to be centralized in the metadata log for various
> > reasons (auditability, visibility, consistency, etc.). Similarly, I am
> not
> > sure about automatically deleting configs in a way that they cannot be
> > recovered. A good property for modern systems is to minimize the number
> of
> > unrecoverable data loss scenarios.
> >
> > Ismael
> >
> > On Wed, Jul 13, 2022 at 3:47 PM David Jacot  >
> > wrote:
> >
> > > Thanks Guozhang. My answers are below:
> > >
> > > > 1) the migration path, especially the last step when clients flip the
> > > flag
> > > > to enable the new protocol, in which we would have a window where
> both
> > > new
> > > > protocols / rpcs and old protocols / rpcs are used by members of the
> same
> > > > group. How the coordinator could "mimic" the old behavior while
> using the
> > > > new protocol is something we need to present about.
> > >
> > > Noted. I just published a new version of KIP which includes more
> > > details about this. See the "Supporting Online Consumer Group Upgrade"
> > > and the "Compatibility, Deprecation, and Migration Plan". I think that
> > > I have to think through a few cases now but the overall idea and
> > > mechanism should be understandable.
> > >
> > > > 2) the usage of topic ids. So far as KIP-516 the topic ids are only
> used
> > > as
> > > > part of RPCs and admin client, but they are not exposed via any
> public
> > > APIs
> > > > to consumers yet. I think the question is, first should we let the
> > > consumer
> > > > client to be maintaining the names -> ids mapping itself to fully
> > > leverage
> > > > on all the augmented existing RPCs and the new RPCs with the topic
> ids;
> > > and
> > > > secondly, should we ever consider exposing the topic ids in the
> consumer
> > > > public APIs as well (both subscribe/assign, as well as 

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

Thanks, Ismael.

> 1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we should
> document the differences in more detail before deciding one way or another.
> That said, if people pass java.util.regex.Pattern, they expect their
> semantics to be honored. If we are doing something different, then we
> should consider adding an overload with our own Pattern class (I don't
> think we'd want to expose re2j's at this point).

Yeah. I do agree with you. I looked a bit more into re2j and the
differences are subtle. Simple regexes will work with both but special
constructs or character classes won't work. I like the idea of having
our own Pattern. That would make things really clear and avoid
compatibility issues. I do agree that we should not expose re2j's
Pattern in our public API.

> 2. Regarding topic ids, any major new protocol should integrate fully with
> it and should handle the topic recreation case correctly. That's the main
> part we need to handle. I agree with David that we'd want to add topic ids
> to the relevant protocols that don't have it yet and that we can probably
> focus on the internals versus adding new APIs to the Java Consumer (unless
> we find that adding new APIs is required for reasonable semantics).

I wonder what we should do with the admin client though. We have a
couple of apis related to offsets there. Do we want to allow users to
fetch, get or delete offsets based on topic ids? On the server side,
we will have to store the offsets based on the topic id. When a topic
is recreated, we will likely have offsets stored with both the old
topic id and the new topic id. From an admin perspective, we can't
really differentiate them without letting the user use the topic id
and providing the topic ids to him as well. One way would be to
disambiguate them based on the group current assignment. I need to
think a little more about this.

> 3. I am still not sure about the coordinator storing the configs. It's
> powerful for configs to be centralized in the metadata log for various
> reasons (auditability, visibility, consistency, etc.). Similarly, I am not
> sure about automatically deleting configs in a way that they cannot be
> recovered. A good property for modern systems is to minimize the number of
> unrecoverable data loss scenarios.

I see your point. One concern that I have regarding storing them in
the controller is that they will never be deleted unless the user does
it. As users consider groups as throwing away resources, it will never
happen. If we draw a parallel with topics, we do delete their configs
when they are deleted. It is also worth pointing out that offsets are
deleted when the group is expired and that is even worse than deleting
configs in my opinion. One way would be to store configs in the
controller and to let the coordinator delete them when a group is
deleted. I suppose that the problem is the same because the configs
are also automatically deleted. Things would be different if groups
were a first class resource in the cluster.

Best,
David

On Fri, Jul 15, 2022 at 1:30 PM David Jacot  wrote:
>
> Thanks Hector! Our goal is to move forward with specialized API
> instead of relying on one generic API. For Connect, we can apply the
> exact same pattern and reuse/share the core implementation on the
> server side. For the schema registry, I think that we should consider
> having a tailored API to do simple membership/leader election.
>
> Best,
> David
>
> On Fri, Jul 15, 2022 at 10:22 AM Ismael Juma  wrote:
> >
> > Three quick comments:
> >
> > 1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we should
> > document the differences in more detail before deciding one way or another.
> > That said, if people pass java.util.regex.Pattern, they expect their
> > semantics to be honored. If we are doing something different, then we
> > should consider adding an overload with our own Pattern class (I don't
> > think we'd want to expose re2j's at this point).
> > 2. Regarding topic ids, any major new protocol should integrate fully with
> > it and should handle the topic recreation case correctly. That's the main
> > part we need to handle. I agree with David that we'd want to add topic ids
> > to the relevant protocols that don't have it yet and that we can probably
> > focus on the internals versus adding new APIs to the Java Consumer (unless
> > we find that adding new APIs is required for reasonable semantics).
> > 3. I am still not sure about the coordinator storing the configs. It's
> > powerful for configs to be centralized in the metadata log for various
> > reasons (auditability, visibility, consistency, etc.). Similarly, I am not
> > sure about automatically deleting configs in a way that they cannot be
> > recovered. A good property for modern systems is to minimize the number of
> > unrecoverable data loss scenarios.
> >
> > Ismael
> >
> > On Wed, Jul 13, 2022 at 3:47 PM David Jacot 
> > wrote:
> >
> > > Thanks Guozhang. My answers a

[GitHub] [kafka-site] StephanZimmermann commented on pull request #406: Add Qudosoft to powered-by

[GitBox]

StephanZimmermann commented on PR #406:
URL: https://github.com/apache/kafka-site/pull/406#issuecomment-1185481331

   Hi @bbejeck , I did the requested rebase. Ready to pull :-) Cheers


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

Hi Sagar,

Thanks for your comments.

1) Yes. That refers to `Assignment#error`. Sure, I can mention it.

2) The idea is to transition C from his current assignment to his
target assignment when he can move to epoch 3. When that happens, the
member assignment is updated and persisted with all its assigned
partitions even if they are not all revoked yet. In other words, the
member assignment becomes the target assignment. This is basically an
optimization to avoid having to write all the changes to the log. The
examples are based on the persisted state so I understand the
confusion. Let me see if I can improve this in the description.

3) Regarding Connect, it could reuse the protocol with a client side
assignor if it fits in the protocol. The assignment is about
topicid-partitions + metadata, could Connect fit into this?

Best,
David

On Fri, Jul 15, 2022 at 1:55 PM Sagar  wrote:
>
> Hi David,
>
> Thanks for the KIP. I just had minor observations:
>
> 1) In the Assignment Error section in Client Side mode Assignment process,
> you mentioned => `In this case, the client side assignor can return an
> error to the group coordinator`. In this case are you referring to the
> Assignor returning an AssignmentError that's listed down towards the end?
> If yes, do you think it would make sense to mention this explicitly here?
>
> 2) In the Case Studies section, I have a slight confusion, not sure if
> others have the same. Consider this step:
>
> When B heartbeats, the group coordinator transitions him to epoch 3 because
> B has no partitions to revoke. It persists the change and reply.
>
>- Group (epoch=3)
>   - A
>   - B
>   - C
>- Target Assignment (epoch=3)
>   - A - partitions=[foo-0]
>   - B - partitions=[foo-2]
>   - C - partitions=[foo-1]
>- Member Assignment
>   - A - epoch=2, partitions=[foo-0, foo-1]
>   - B - epoch=3, partitions=[foo-2]
>   - C - epoch=3, partitions=[foo-1]
>
> When C heartbeats, it transitions to epoch 3 but cannot get foo-1 yet.
>
> Here,it's mentioned that member C can't get the foo-1 partition yet, but
> based on the description above, it seems it already has it. Do you think it
> would be better to remove it and populate it only when it actually gets it?
> I see this in a lot of other places, so have I understood it incorrectly ?
>
>
> Regarding connect , it might be out of scope of this discussion, but from
> what I understood it would probably be running in client side assignor mode
> even on the new rebalance protocol as it has its own Custom Assignors(Eager
> and IncrementalCooperative).
>
> Thanks!
>
> Sagar.
>
>
>
>
>
>
> On Fri, Jul 15, 2022 at 5:00 PM David Jacot 
> wrote:
>
> > Thanks Hector! Our goal is to move forward with specialized API
> > instead of relying on one generic API. For Connect, we can apply the
> > exact same pattern and reuse/share the core implementation on the
> > server side. For the schema registry, I think that we should consider
> > having a tailored API to do simple membership/leader election.
> >
> > Best,
> > David
> >
> > On Fri, Jul 15, 2022 at 10:22 AM Ismael Juma  wrote:
> > >
> > > Three quick comments:
> > >
> > > 1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we
> > should
> > > document the differences in more detail before deciding one way or
> > another.
> > > That said, if people pass java.util.regex.Pattern, they expect their
> > > semantics to be honored. If we are doing something different, then we
> > > should consider adding an overload with our own Pattern class (I don't
> > > think we'd want to expose re2j's at this point).
> > > 2. Regarding topic ids, any major new protocol should integrate fully
> > with
> > > it and should handle the topic recreation case correctly. That's the main
> > > part we need to handle. I agree with David that we'd want to add topic
> > ids
> > > to the relevant protocols that don't have it yet and that we can probably
> > > focus on the internals versus adding new APIs to the Java Consumer
> > (unless
> > > we find that adding new APIs is required for reasonable semantics).
> > > 3. I am still not sure about the coordinator storing the configs. It's
> > > powerful for configs to be centralized in the metadata log for various
> > > reasons (auditability, visibility, consistency, etc.). Similarly, I am
> > not
> > > sure about automatically deleting configs in a way that they cannot be
> > > recovered. A good property for modern systems is to minimize the number
> > of
> > > unrecoverable data loss scenarios.
> > >
> > > Ismael
> > >
> > > On Wed, Jul 13, 2022 at 3:47 PM David Jacot  > >
> > > wrote:
> > >
> > > > Thanks Guozhang. My answers are below:
> > > >
> > > > > 1) the migration path, especially the last step when clients flip the
> > > > flag
> > > > > to enable the new protocol, in which we would have a window where
> > both
> > > > new
> > > > > protocols / rpcs and old protocols / rpcs are used by members of the
> > same
> 

Re: [DISCUSS] Apache Kafka 3.2.1 release

[David Arthur]

Here is the release plan for 3.2.1
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.1

I am working on getting clarity on the one open blocker. Once that is
resolved (or rescheduled for a future release), I will build the first
release candidate.

-David

On Thu, Jul 14, 2022 at 3:10 AM Luke Chen  wrote:

> +1, Thanks David!
>
> On Thu, Jul 14, 2022 at 1:16 PM David Jacot  wrote:
>
> > +1. Thanks David.
> >
> > Le mer. 13 juil. 2022 à 23:43, José Armando García Sancio
> >  a écrit :
> >
> > > +1. Thanks for volunteering David.
> > >
> > > --
> > > -José
> > >
> >
>


-- 
David Arthur

Re: [DISCUSS] KIP-848: The Next Generation of the Consumer Rebalance Protocol

[David Jacot]

I'll be away from July 18th to August 8th with limited access to my
emails so I will address new comments and questions when I come back.

Cheers,
David

On Fri, Jul 15, 2022 at 2:16 PM David Jacot  wrote:
>
> Hi Sagar,
>
> Thanks for your comments.
>
> 1) Yes. That refers to `Assignment#error`. Sure, I can mention it.
>
> 2) The idea is to transition C from his current assignment to his
> target assignment when he can move to epoch 3. When that happens, the
> member assignment is updated and persisted with all its assigned
> partitions even if they are not all revoked yet. In other words, the
> member assignment becomes the target assignment. This is basically an
> optimization to avoid having to write all the changes to the log. The
> examples are based on the persisted state so I understand the
> confusion. Let me see if I can improve this in the description.
>
> 3) Regarding Connect, it could reuse the protocol with a client side
> assignor if it fits in the protocol. The assignment is about
> topicid-partitions + metadata, could Connect fit into this?
>
> Best,
> David
>
> On Fri, Jul 15, 2022 at 1:55 PM Sagar  wrote:
> >
> > Hi David,
> >
> > Thanks for the KIP. I just had minor observations:
> >
> > 1) In the Assignment Error section in Client Side mode Assignment process,
> > you mentioned => `In this case, the client side assignor can return an
> > error to the group coordinator`. In this case are you referring to the
> > Assignor returning an AssignmentError that's listed down towards the end?
> > If yes, do you think it would make sense to mention this explicitly here?
> >
> > 2) In the Case Studies section, I have a slight confusion, not sure if
> > others have the same. Consider this step:
> >
> > When B heartbeats, the group coordinator transitions him to epoch 3 because
> > B has no partitions to revoke. It persists the change and reply.
> >
> >- Group (epoch=3)
> >   - A
> >   - B
> >   - C
> >- Target Assignment (epoch=3)
> >   - A - partitions=[foo-0]
> >   - B - partitions=[foo-2]
> >   - C - partitions=[foo-1]
> >- Member Assignment
> >   - A - epoch=2, partitions=[foo-0, foo-1]
> >   - B - epoch=3, partitions=[foo-2]
> >   - C - epoch=3, partitions=[foo-1]
> >
> > When C heartbeats, it transitions to epoch 3 but cannot get foo-1 yet.
> >
> > Here,it's mentioned that member C can't get the foo-1 partition yet, but
> > based on the description above, it seems it already has it. Do you think it
> > would be better to remove it and populate it only when it actually gets it?
> > I see this in a lot of other places, so have I understood it incorrectly ?
> >
> >
> > Regarding connect , it might be out of scope of this discussion, but from
> > what I understood it would probably be running in client side assignor mode
> > even on the new rebalance protocol as it has its own Custom Assignors(Eager
> > and IncrementalCooperative).
> >
> > Thanks!
> >
> > Sagar.
> >
> >
> >
> >
> >
> >
> > On Fri, Jul 15, 2022 at 5:00 PM David Jacot 
> > wrote:
> >
> > > Thanks Hector! Our goal is to move forward with specialized API
> > > instead of relying on one generic API. For Connect, we can apply the
> > > exact same pattern and reuse/share the core implementation on the
> > > server side. For the schema registry, I think that we should consider
> > > having a tailored API to do simple membership/leader election.
> > >
> > > Best,
> > > David
> > >
> > > On Fri, Jul 15, 2022 at 10:22 AM Ismael Juma  wrote:
> > > >
> > > > Three quick comments:
> > > >
> > > > 1. Regarding java.util.regex.Pattern vs com.google.re2j.Pattern, we
> > > should
> > > > document the differences in more detail before deciding one way or
> > > another.
> > > > That said, if people pass java.util.regex.Pattern, they expect their
> > > > semantics to be honored. If we are doing something different, then we
> > > > should consider adding an overload with our own Pattern class (I don't
> > > > think we'd want to expose re2j's at this point).
> > > > 2. Regarding topic ids, any major new protocol should integrate fully
> > > with
> > > > it and should handle the topic recreation case correctly. That's the 
> > > > main
> > > > part we need to handle. I agree with David that we'd want to add topic
> > > ids
> > > > to the relevant protocols that don't have it yet and that we can 
> > > > probably
> > > > focus on the internals versus adding new APIs to the Java Consumer
> > > (unless
> > > > we find that adding new APIs is required for reasonable semantics).
> > > > 3. I am still not sure about the coordinator storing the configs. It's
> > > > powerful for configs to be centralized in the metadata log for various
> > > > reasons (auditability, visibility, consistency, etc.). Similarly, I am
> > > not
> > > > sure about automatically deleting configs in a way that they cannot be
> > > > recovered. A good property for modern systems is to minimize the number
> > > of
> > > > unrecoverable dat