[
https://issues.apache.org/jira/browse/KAFKA-13612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17482115#comment-17482115
]
Dominic Evans edited comment on KAFKA-13612 at 1/25/22, 9:25 PM:
-----------------------------------------------------------------
[~hachikuji] I have no affiliation with Burrow, but I can wager a guess as to
why they choose to read from _consumer_offsets directly. If you're writing any
software to monitor the lag of Kafka Consumer groups, it's much more efficient
to be asynchronously driven by the changes to the {_}`{_}_consumer_offsets`
partitions than it is to repeatedly poll the Kafka API, cache results and
manage evictions yourself.
Option 1: read from __consumer_offsets and parse the data format yourself
Option 2:
* poll every broker in the cluster with ListGroupsRequests
* send FindCoordinator requests for every group and put them in batches
* send DescribeGroupRequest to each coordinator with the batches of consumer
groups
* read the results out of the DescribeGroupResponses and go round the loop
again
was (Author: dnwe):
[~hachikuji] I have no affiliation with Burrow, but I can wager a guess as to
why they choose to read from __consumer_offsets directly. If you're writing any
software to monitor the lag of Kafka Consumer groups, it's much more efficient
to be asynchronously driven by the changes to the `__consumer_offsets`
partitions than it is to repeatedly poll the Kafka API, cache results and
manage evictions yourself.
Option 1: read from __consumer_offsets and parse the data format yourself
Option 2:
* poll every broker in the cluster with ListGroupsRequests
* send FindCoordinator requests for every group and put them in batches
* send DescribeGroupRequest to each coordinator with the batches of consumer
groups
* read the results out of the DescribeGroupResponses and go round the loop
again
> internal topics won't be created in metadataRequest when
> auto.create.topics.enable=false
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-13612
> URL: https://issues.apache.org/jira/browse/KAFKA-13612
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 2.8.0, 3.0.0
> Reporter: Luke Chen
> Assignee: Luke Chen
> Priority: Major
>
> In KAFKA-9751, when create internal topics through FindCoordinator or
> Metadata request, we route the topic creation request to the controller
> instead of handling by itself. We change logic in
> `KafkaApis#getTopicMetadata`, and make the internal topic won't get created
> when "auto.create.topics.enable=false`.
> h4.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
