[
https://issues.apache.org/jira/browse/KAFKA-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14009835#comment-14009835
]
Guozhang Wang commented on KAFKA-170:
-------------------------------------
Claude, in 0.9 we are going to rewrite our consumer client and it will use
non-blocking poll APIs, do you want to check its design and see if it satisfies
your use case?
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
> Support for non-blocking polling on multiple streams
> ----------------------------------------------------
>
> Key: KAFKA-170
> URL: https://issues.apache.org/jira/browse/KAFKA-170
> Project: Kafka
> Issue Type: Sub-task
> Components: core
> Affects Versions: 0.8.0
> Reporter: Jay Kreps
> Priority: Critical
> Labels: replication
>
> Currently we provide a blocking iterator in the consumer. This is a good
> mechanism for consuming data from a single topic, but is limited as a
> mechanism for polling multiple streams.
> For example if one wants to implement a non-blocking union across multiple
> streams this is hard to do because calls may block indefinitely. A similar
> situation arrises if trying to implement a streaming join of between two
> streams.
> I would propose two changes:
> 1. Implement a next(timeout) interface on KafkaMessageStream. This will
> easily handle some simple cases with minimal change. This handles certain
> limited cases nicely and is easy to implement, but doesn't actually cover the
> two cases above.
> 2. Add an interface to poll streams.
> I don't know the best approach for the later api, but it is important to get
> it right. One option would be to add a
> ConsumerConnector.drainTopics("topic1", "topic2", ...) which blocks until
> there is at least one message and then returns a list of triples (topic,
> partition, message).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
