[
https://issues.apache.org/jira/browse/KAFKA-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15879418#comment-15879418
]
Jason Gustafson commented on KAFKA-1895:
----------------------------------------
It's an interesting idea, but it's also a big change to the API, right? Having
two ways to poll the data and with differing semantics would make the consumer
more confusing (we've had a tough enough time explaining how the current API
works). I think we should be clear about the problem we are trying to solve. It
sounds like what you are looking for is to avoid copying the message data.
Maybe a simpler way to achieve that would be to have a new Deserializer type
which works with byte buffers instead of byte arrays? One challenge is that the
consumer would probably like to control its own memory pool (see
https://cwiki.apache.org/confluence/display/KAFKA/KIP-81%3A+Bound+Fetch+memory+usage+in+the+consumer),
so it might not be a great idea to give users direct access to the underlying
buffers.
> Investigate moving deserialization and decompression out of KafkaConsumer
> -------------------------------------------------------------------------
>
> Key: KAFKA-1895
> URL: https://issues.apache.org/jira/browse/KAFKA-1895
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Jay Kreps
>
> The consumer implementation in KAFKA-1760 decompresses fetch responses and
> deserializes them into ConsumerRecords which are then handed back as the
> result of poll().
> There are several downsides to this:
> 1. It is impossible to scale serialization and decompression work beyond the
> single thread running the KafkaConsumer.
> 2. The results can come back during the processing of other calls such as
> commit() etc which can result in caching these records a little longer.
> An alternative would be to have ConsumerRecords wrap the actual compressed
> serialized MemoryRecords chunks and do the deserialization during iteration.
> This way you could scale this over a thread pool if needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
