[
https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15566191#comment-15566191
]
Michael Armbrust commented on SPARK-17344:
------------------------------------------
I think the fact that CDH is still distributing 0.9 is a pretty convincing
argument.
I'm also not convinced its a bad idea to and speak the protocol directly. Our
use case ends up being significantly simpler than most other consumer
implementations since we have the opportunity to do global coordination on the
driver. As such, we'd really only to correctly to handle two types of
requests:
[TopicMetadataRequest|https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-TopicMetadataRequest]
and
[FetchRequest|https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-FetchRequest].
- The variations here across versions are minimal for these messages.
- We could avoid have N different artifacts for N versions of kafka
- We could remove the complexity of caching consumers on executors (though
still set preferred locations to encourage collocation).
- We could avoid extra copies of the payload when going from the kafka library
into tungsten.
I agree we shouldn't make this decision lightly, but looking at our past
experience supporting multiple versions of Hadoop/Hive as transparently as
possible, I think this could be a big boost for adoption.
> Kafka 0.8 support for Structured Streaming
> ------------------------------------------
>
> Key: SPARK-17344
> URL: https://issues.apache.org/jira/browse/SPARK-17344
> Project: Spark
> Issue Type: Sub-task
> Components: Streaming
> Reporter: Frederick Reiss
>
> Design and implement Kafka 0.8-based sources and sinks for Structured
> Streaming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
