Re: SQL + streaming

Aah, glad you found it out. TD On Tue, Jul 15, 2014 at 7:52 PM, hsy...@gmail.com wrote: > Thanks Tathagata, we actually found the problem. I created SQLContext and > StreamContext from different SparkContext. But thanks for your help > > Best, > Siyuan > > > On Tue, Jul 15, 2014 at 6:53 PM, T

Re: SQL + streaming

Thanks Tathagata, we actually found the problem. I created SQLContext and StreamContext from different SparkContext. But thanks for your help Best, Siyuan On Tue, Jul 15, 2014 at 6:53 PM, Tathagata Das wrote: > Oh yes, we have run sql, streaming and mllib all together. > > You can take a look

Re: SQL + streaming

Oh yes, we have run sql, streaming and mllib all together. You can take a look at the demo that DataBricks gave at the spark summit. I think I get the problem is. Sql("") returns a RDD, and println(rdd) prints only the RDD's name. And rdd.foreach(println) prints

Re: SQL + streaming

By the way, have you ever run SQL and stream together? Do you know any example that works? Thanks! On Tue, Jul 15, 2014 at 4:28 PM, hsy...@gmail.com wrote: > Hi Tathagata, > > I could see the output of count, but no sql results. Run in standalone is > meaningless for me and I just run in my loc

Re: SQL + streaming

Hi Tathagata, I could see the output of count, but no sql results. Run in standalone is meaningless for me and I just run in my local single node yarn cluster. Thanks On Tue, Jul 15, 2014 at 12:48 PM, Tathagata Das wrote: > Could you run it locally first to make sure it works, and you see outp

Re: SQL + streaming

Could you run it locally first to make sure it works, and you see output? Also, I recommend going through the previous step-by-step approach to narrow down where the problem is. TD On Mon, Jul 14, 2014 at 9:15 PM, hsy...@gmail.com wrote: > Actually, I deployed this on yarn cluster(spark-submit

Re: SQL + streaming

Actually, I deployed this on yarn cluster(spark-submit) and I couldn't find any output from the yarn stdout logs On Mon, Jul 14, 2014 at 6:25 PM, Tathagata Das wrote: > Can you make sure you are running locally on more than 1 local cores? You > could set the master in the SparkConf as conf.setM

Re: SQL + streaming

Can you make sure you are running locally on more than 1 local cores? You could set the master in the SparkConf as conf.setMaster("local[4]"). Then see if there are jobs running on every batch of data in the Spark web ui (running on localhost:4040). If you still dont get any output, try first simpl

Re: SQL + streaming

No errors but no output either... Thanks! On Mon, Jul 14, 2014 at 4:59 PM, Tathagata Das wrote: > Could you elaborate on what is the problem you are facing? Compiler error? > Runtime error? Class-not-found error? Not receiving any data from Kafka? > Receiving data but SQL command throwing error

Re: SQL + streaming

Could you elaborate on what is the problem you are facing? Compiler error? Runtime error? Class-not-found error? Not receiving any data from Kafka? Receiving data but SQL command throwing error? No errors but no output either? TD On Mon, Jul 14, 2014 at 4:06 PM, hsy...@gmail.com wrote: > Hi Al