[
https://issues.apache.org/jira/browse/SPARK-17344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15567321#comment-15567321
]
Michael Armbrust commented on SPARK-17344:
------------------------------------------
These are good questions. A few thoughts:
bq. How long would it take CDH to distribute 0.10 if there was a compelling
Spark client for it?
Even if they were going to release kafka 0.10 in CDH yesterday, my experience
is that many will take a long time for people to upgrade. We spent a fair
amount of effort on multi-version compatibility for Hive in Spark SQL and it
was great boost for adoption. I think this could be the same thing.
bq. How are you going to handle SSL? You can't avoid the complexity of caching
consumers if you still want the benefits of prefetching, and doing an SSL
handshake for every batch will kill performance if they aren't cached.
An option here would be to use the internal client directly. This way we can
leverage all the work that they did to support SSL, etc yet make it speak
specific versions of the protocol as we need. I did a [really rough
prototype|https://gist.github.com/marmbrus/7d116b0a9672337497ddfccc0657dbf0]
using the APIs described above and it is not that much code. There is clearly
a lot more we'd need to do, but I think we should strongly consider this option.
Caching connections to the specific brokers should probably still be
implemented for the reasons you describe (and this is already handled by the
internal client). An advantage here is you'd actually be able to share
connections across queries without running into correctness problems.
> Kafka 0.8 support for Structured Streaming
> ------------------------------------------
>
> Key: SPARK-17344
> URL: https://issues.apache.org/jira/browse/SPARK-17344
> Project: Spark
> Issue Type: Sub-task
> Components: Streaming
> Reporter: Frederick Reiss
>
> Design and implement Kafka 0.8-based sources and sinks for Structured
> Streaming.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
