[
https://issues.apache.org/jira/browse/KAFKA-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295281#comment-16295281
]
Ryan Leslie commented on KAFKA-6287:
------------------------------------
Thanks for reviewing. I agree that the protocol type shouldn't be cleared. A
group could become empty temporarily during an outage or user upgrade and it
doesn't make sense to lose information about the group type in those cases.
It's probably best to keep everything explicit, and as you noted the
DeleteGroup API is a step in that direction. For the record, I currently
simulate deleting groups by sending OffsetCommitRequest with a retention time
of 0 and keeping offsets.retention.check.interval.ms small,
but would much prefer the explicit request!
As far as having groups default to "consumer" I think ensuring it is set in the
metadata makes sense (option 1). Again, this will keep everything explicit, and
this way groups will always have an associated type. I think it's fine to
hardcode the default as "consumer" for now, since there probably aren't enough
other use cases of group management to warrant making it configurable. The
setting could always be added in the future.
> Inconsistent protocol type for empty consumer groups
> ----------------------------------------------------
>
> Key: KAFKA-6287
> URL: https://issues.apache.org/jira/browse/KAFKA-6287
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 1.0.0
> Reporter: Ryan Leslie
> Assignee: Jason Gustafson
> Priority: Minor
>
> When a consumer is created for a new group, the group metadata's protocol
> type is set to 'consumer' and this is stored both in __consumer_offsets as
> well as in the coordinator's local cache.
> If the consumer leaves the group and the group becomes empty, ListGroups
> requests will continue to show the group as type 'consumer', and as such
> kafka-consumer-groups.sh will show it via --list.
> However, if the coordinator (broker) node is killed and a new coordinator is
> elected, when the GroupMetadataManager loads the group from
> __consumer_offsets into its cache, it will not set the protocolType if there
> are no active consumers. As a result, the group's protocolType will now
> become the empty string (UNKNOWN_PROTOCOL_TYPE), and kafka-consumer-groups.sh
> will no longer show the group.
> Ideally bouncing a broker should not result in the group's protocol changing.
> protocolType can be set in GroupMetadataManager.readGroupMessageValue() to
> always reflect what's present in the persistent metadata (__consumer_offsets)
> regardless if there are active members.
> In general, things can get confusing when distinguishing between 'consumer'
> and non-consumer groups. For example, DescribeGroups and OffsetFetchRequest
> does not filter on protocol type, so it's possible for
> kafka-consumer-groups.sh to describe groups (--describe) without actually
> being able to list them. In the protocol guide, OffsetFetchRequest /
> OffsetCommitRequest have their parameters listed as 'ConsumerGroup', but in
> reality these can be used for groups of unknown type as well. For instance, a
> consumer group can be copied by finding a coordinator
> (GroupCoordinatorRequest / FindCoordinatorRequest) and sending an
> OffsetCommitRequest. The group will be auto-created with an empty protocol.
> Although this is arguably correct, the group will now exist but not be a
> proper 'consumer' group until later when there is a JoinGroupRequest. Again,
> this can be confusing as far as categorization / visibility of the group is
> concerned. A group can instead be copied more directly by creating a consumer
> and calling commitSync (as kafka-consumer-groups.sh), but this does involve
> extra connections / requests and so is a little slower when trying to keep a
> large number of groups in sync in real-time across clusters.
> If we want to make it easier to keep consumer groups consistent, options
> include allowing groups to be explicitly created with a protocol type instead
> of only lazily, or for groups created outside of JoinGroupRequest the
> coordinator can gain a broker config to set a default protocol type for
> groups. This would default to 'consumer'. This sort of setting might be
> confusing for users though, since implicitly created groups is certainly not
> the norm.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
