[
https://issues.apache.org/jira/browse/KAFKA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011513#comment-14011513
]
Jay Kreps commented on KAFKA-1316:
----------------------------------
[~junrao] Here is how I think of the boundary of responsibilities:
1. The org.apache.kafka.common.network package only knows about the
transmission of byte arrays.
2. The org.apache.kafka.common.protocol.types knows about defining generic
message formats (nothing about I/O)
3. The org.apache.kafka.common.protocol package defines the kafka protocol with
no reference to I/O
4. These get wrapped up into the org.apache.clients package. This level is
aware of (1), (2), and (3) however it has no logic specific to a particular
client (producer, consumer, admin).
Thinking about it this way it makes sense to support cluster metadata in the
general kafka client. This is the basic data of which nodes there are, what
their hosts and ports are, and what partitions they host.
It doesn't make sense to handle consumer metadata here as this is totally
specific to the consumer.
I considered (2) as a solution to the blocking fetch problem Neha and I
discussed. But I think this is not ideal as it means you have to poll() over
two sets of sockets. Since both polls are blocking this means you need
additional threads and need to somehow coordinate the offset commit with the
client consumption.
> Refactor Sender
> ---------------
>
> Key: KAFKA-1316
> URL: https://issues.apache.org/jira/browse/KAFKA-1316
> Project: Kafka
> Issue Type: Sub-task
> Components: producer
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Attachments: KAFKA-1316.patch
>
>
> Currently most of the logic of the producer I/O thread is in Sender.java.
> However we will need to do a fair number of similar things in the new
> consumer. Specifically:
> - Track in-flight requests
> - Fetch metadata
> - Manage connection lifecycle
> It may be possible to refactor some of this into a helper class that can be
> shared with the consumer. This will require some detailed thought.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
