[
https://issues.apache.org/jira/browse/SPARK-16407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15491784#comment-15491784
]
holdenk commented on SPARK-16407:
---------------------------------
It's true it doesn't work in SQL - but I don't think the current stream writer
interface works so well in SQL anyways.
While its true this doesn't yet expose a Python API - thats generally true for
structured streaming. Once we do add a Python API its possible one could wrap
Scala sinks that do custom callbacks to Python (similar to how the current
Python streaming API works) or otherwise provide wrappers for JVM sinks as is
the general process for a lot of PySpark.
I'm not advocating for this to replace the string based API but to compliment
it to allow more flexibility for users (as demonstrated in the provided test).
> Allow users to supply custom StreamSinkProviders
> ------------------------------------------------
>
> Key: SPARK-16407
> URL: https://issues.apache.org/jira/browse/SPARK-16407
> Project: Spark
> Issue Type: Improvement
> Components: Streaming
> Reporter: holdenk
>
> The current DataStreamWriter allows users to specify a class name as format,
> however it could be easier for people to directly pass in a specific provider
> instance - e.g. for user equivalent of ForeachSink or other sink with
> non-string parameters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
