Re: Trying to run SparkSQL over Spark Streaming

Hi, I'm also trying to use the insertInto method, but end up getting the "assertion error" Is there any workaround to this?? rgds -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trying-to-run-SparkSQL-over-Spark-Streaming-tp12530p21316

Re: Trying to run SparkSQL over Spark Streaming

d to table. So i guess inserting into parquet tables is also not supported? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trying-to-run-SparkSQL-over-Spark-Streaming-tp12530p13004.html Sent from the Apache Spark User List mailing list archi

Re: Trying to run SparkSQL over Spark Streaming

block, >>> or -- as you experienced -- the table name will be unknown" >>> >>> Since this is the case then is there any way to run join over data >>> received >>> from two different streams? >>> >> >> Couldn't you do dstrea

Re: Trying to run SparkSQL over Spark Streaming

LECT * FROM dstream1 JOIN dstream2 WHERE ..."? I don't know if that is possible. Doesn't seem easy to me; I don't think that's doable with the current codebase... Tobias -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trying-to-run

Re: Trying to run SparkSQL over Spark Streaming

Hi again, On Tue, Aug 26, 2014 at 10:13 AM, Tobias Pfeiffer wrote: > > On Mon, Aug 25, 2014 at 7:11 PM, praveshjain1991 < > praveshjain1...@gmail.com> wrote: >> >> "If you want to issue an SQL statement on streaming data, you must have >> both >> the registerAsTable() and the sql() call *within*

Re: Trying to run SparkSQL over Spark Streaming

Hi, On Mon, Aug 25, 2014 at 7:11 PM, praveshjain1991 wrote: > > "If you want to issue an SQL statement on streaming data, you must have > both > the registerAsTable() and the sql() call *within* the foreachRDD(...) > block, > or -- as you experienced -- the table name will be unknown" > > Since

Re: Trying to run SparkSQL over Spark Streaming

known" Since this is the case then is there any way to run join over data received from two different streams? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trying-to-run-SparkSQL-over-Spark-Streaming-tp12530p12739.html Sent from the Apache

RE: Trying to run SparkSQL over Spark Streaming

To: u...@spark.incubator.apache.org Subject: Re: Trying to run SparkSQL over Spark Streaming Oh right. Got it. Thanks Also found this link on that discussion: https://github.com/thunderain-project/StreamSQL Does this provide more features than Spark? -- View this message in context: http://apache-spark

Re: Trying to run SparkSQL over Spark Streaming

Oh right. Got it. Thanks Also found this link on that discussion: https://github.com/thunderain-project/StreamSQL Does this provide more features than Spark? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trying-to-run-SparkSQL-over-Spark-Streaming

Re: Trying to run SparkSQL over Spark Streaming

Hi, On Thu, Aug 21, 2014 at 3:11 PM, praveshjain1991 wrote: > > The part that you mentioned "*/the variable `result ` is of type > DStream[Row]. That is, the meta-information from the SchemaRDD is lost and, > from what I understand, there is then no way to learn about the column > names > of the

Re: Trying to run SparkSQL over Spark Streaming

from the SchemaRDD is lost and, from what I understand, there is then no way to learn about the column names of the returned data, as this information is only encoded in the SchemaRDD/*" Why is this bad?? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Trying-to

Re: Trying to run SparkSQL over Spark Streaming

Hi, On Thu, Aug 21, 2014 at 2:19 PM, praveshjain1991 wrote: > > Using Spark SQL with batch data works fine so I'm thinking it has to do > with > how I'm calling streamingcontext.start(). Any ideas what is the issue? Here > is the code: > Please have a look at http://apache-spark-user-list.100

Trying to run SparkSQL over Spark Streaming

rim.toInt)).registerAsTable("data")) // lines.foreachRDD(rdd=>rdd.foreach(println)) val teenagers = sqc.sql("SELECT name FROM data WHERE age >= 13 AND age <= 19") ssc.start() ssc.awaitTermination() } } Any suggestions welcome. Thanks. -- View this message