[
https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15495006#comment-15495006
]
Reynold Xin commented on SPARK-16407:
-------------------------------------
The source/sink interface currently depends on RDDs, doesn't it? In that case,
it has two issues:
1. The RDD interface is wide and difficult to stabilize across versions. This
is similar to point 1 in https://issues.apache.org/jira/browse/SPARK-15689.
Ideally, a source/sink implementation created for Spark 2.x should work in
Spark 10.x, assuming the JVM is still around.
2. It is difficult to swap in/out a different execution engine.
Actually I'm going to move the above into SPARK-10815 and just continue the
discussion there.
> Allow users to supply custom StreamSinkProviders
> ------------------------------------------------
>
> Key: SPARK-16407
> URL: https://issues.apache.org/jira/browse/SPARK-16407
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: holdenk
>
> The current DataStreamWriter allows users to specify a class name as format,
> however it could be easier for people to directly pass in a specific provider
> instance - e.g. for user equivalent of ForeachSink or other sink with
> non-string parameters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
