Structured Streaming is not designed to integrate with receivers. The sources in Structured Streaming are designed for providing stronger fault-tolerance guarantees by precisely tracking records by their offsets (e.g. Kafka offsets). This is different from the Receiver APIs which did not require records to have trackable offsets, but didnt provide exactly-once guarantees either.
If you really want to use Structured streaming, you could try writing a simpler application that blindly forwards data from your custom source to Kafka, and then run Structured Streaming on Kafka. We have added very good support for reading from and writing to Kafka in Spark. https://databricks.com/blog/2017/04/26/processing-data-in-apache-kafka-with-structured-streaming-in-apache-spark-2-2.html On Thu, May 4, 2017 at 12:43 AM, kant kodali <kanth...@gmail.com> wrote: > Hi All, > > I have a Custom Receiver that implements onStart() and OnStop Methods of > the Receiver class and I am trying to figure out how to integrate with > SparkSession since I want to do stateful analytics using Structured > Streaming. I couldn't find it in the docs. any idea? When I was doing > stateless streaming I was calling > javaStreamingContext.receiverStream(CustomReceiver) > and it worked fine but now I am trying to do Structured Streaming using > the same custom receiver and I couldn't find how to integrate with > SparkSession. > > Thanks! >
