Here is an example of how you can do. Lets say "myDStream" contains the
data that you may want to asynchornously query, say using, Spark SQL.
val sqlContext = new SqlContext(streamingContext.sparkContext)
myDStream.foreachRDD { rdd => // rdd is a RDD of case class
sqlContext.registerRDDAsTab
Thanks for the reply, I am trying to setup a streaming as a service
approach, using the framework that is used for spark-jobserver. for that I
would need to handle asynchronous operations that are initiated from
outside of the stream. Do you think it is not possible?
On Fri Feb 13 2015 at 10:14:1
Sure it's possible, but you would use Streaming to update some shared
state, and create another service that accessed that shared state too.
On Fri, Feb 13, 2015 at 11:57 AM, Tamas Jambor wrote:
> Thanks for the reply, I am trying to setup a streaming as a service
> approach, using the framework
You call awaitTermination() in the main thread, and indeed it blocks
there forever. From there Spark Streaming takes over, and is invoking
the operations you set up. Your operations have access to the data of
course. That's the model; you don't make external threads that reach
in to Spark Streaming
