[
https://issues.apache.org/jira/browse/KAFKA-2275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634404#comment-14634404
]
Ashish K Singh commented on KAFKA-2275:
---------------------------------------
[~hachikuji] thanks for your response. I was not planning to have the polling
part done as part of this JIRA, as it was not evident from the title. However,
given that we want this for wildcard subscription, below is what I suggest.
Suggested changes:
1. If we have wildcard subscription, have NetworkClient to always get metadata
for all topics.
2. On receiving metadata response, check all topics added to metadata by
consumer are present in response's cluster object. If not, then we probably
want those topics to be created, so send a metadata request for those missing
topics specifically. We should wait for this metadata request to complete.
3. Network client will keep metadata for all topics in cluster.
4. Network client will upgrade metadata's cluster with a pruned copy of cluster
that will only have information related to the topics that metadata has.
Pros:
1. No change in current behavior, except that while creating new topics it
might take some extra cycles.
2. No new background thread required, no extra logic required to keep polling
for metadata with info on all partitions.
Cons:
1. Creating topics via. auto create topics might take longer, should not be
significant though.
2. Two copies of cluster objects will be maintained. One with network client,
which will have info on all topics and one with metadata that will have info on
topics that are explicitly added to metadata by consumer. We can do away with
just one copy with a few more loc in metadata class, but concerns were
expressed on adding more info to a stateful object i.e., metadata. I am fine
either ways.
Let me know.
> Add a ListTopics() API to the new consumer
> ------------------------------------------
>
> Key: KAFKA-2275
> URL: https://issues.apache.org/jira/browse/KAFKA-2275
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Guozhang Wang
> Assignee: Ashish K Singh
> Priority: Critical
> Fix For: 0.8.3
>
> Attachments: KAFKA-2275.patch, KAFKA-2275_2015-07-17_21:39:27.patch,
> KAFKA-2275_2015-07-20_10:44:19.patch
>
>
> With regex subscription like
> {code}
> consumer.subscribe("topic*")
> {code}
> The partition assignment is automatically done at the Kafka side, while there
> are some use cases where consumers want regex subscriptions but not
> Kafka-side partition assignment, rather with their own specific partition
> assignment. With ListTopics() they can periodically check for topic list
> changes and specifically subscribe to the partitions of the new topics.
> For implementation, it involves sending a TopicMetadataRequest to a random
> broker and parse the response.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
