Thanks for the reply.
Ya it doesn't seem doable straight away. Someone suggested this
/For each of your streams, first create an emty RDD that you register as a
table, obtaining an empty table. For your example, let's say you call it
"allTeenagers".
Then, for each of your queries, use SchemaRDD's insertInto method to add the
result to that table:
teenagers.insertInto("allTeenagers")
If you do this with both your streams, creating two separate accumulation
tables, you can then join them using a plain old SQL query.
/
So I was trying it but can't seem to use the insertInto method in the
correct way. Something like:
var p1 = Person("Hari",22)
val rdd1 = sc.parallelize(Array(p1))
rdd1.registerAsTable("data")
var p2 = Person("sagar", 22)
var rdd2 = sc.parallelize(Array(p2))
rdd2.insertInto("data")
is giving the error : "java.lang.AssertionError: assertion failed: No plan
for InsertIntoTable Map(), false"
Any thoughts?
Thanks
Hi again,
On Tue, Aug 26, 2014 at 10:13 AM, Tobias Pfeiffer <tgp@> wrote:
>
> On Mon, Aug 25, 2014 at 7:11 PM, praveshjain1991 <
> praveshjain1991@> wrote:
>>
>> "If you want to issue an SQL statement on streaming data, you must have
>> both
>> the registerAsTable() and the sql() call *within* the foreachRDD(...)
>> block,
>> or -- as you experienced -- the table name will be unknown"
>>
>> Since this is the case then is there any way to run join over data
>> received
>> from two different streams?
>>
>
> Couldn't you do dstream1.join(dstream2).foreachRDD(...)?
>
Ah, I guess you meant something like "SELECT * FROM dstream1 JOIN dstream2
WHERE ..."? I don't know if that is possible. Doesn't seem easy to me; I
don't think that's doable with the current codebase...
Tobias
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Trying-to-run-SparkSQL-over-Spark-Streaming-tp12530p12812.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
