; *To:* user@spark.apache.org
> *Subject:* Re: Some question about SQL and streaming
>
>
>
> Hi,
>
>
>
> On Fri, Jul 11, 2014 at 11:38 AM, Shao, Saisai
> wrote:
>
> Actually we have a POC project which shows the power of combining Spark
> Streaming and Cataly
...@preferred.jp]
Sent: Friday, July 11, 2014 10:47 AM
To: user@spark.apache.org
Subject: Re: Some question about SQL and streaming
Hi,
On Fri, Jul 11, 2014 at 11:38 AM, Shao, Saisai
mailto:saisai.s...@intel.com>> wrote:
Actually we have a POC project which shows the power of combining Spark
Streami
Hi,
On Fri, Jul 11, 2014 at 11:38 AM, Shao, Saisai
wrote:
> Actually we have a POC project which shows the power of combining Spark
> Streaming and Catalyst, it can manipulate SQL on top of Spark Streaming and
> get SchemaDStream. You can take a look at it:
> https://github.com/thunderain-proje
[mailto:tathagata.das1...@gmail.com]
Sent: Friday, July 11, 2014 10:17 AM
To: user@spark.apache.org
Subject: Re: Some question about SQL and streaming
Yeah, the right solution is to have something like SchemaDStream, where the
schema of all the schemaRDD generated by it can be stored. Something I really
would like
Yeah, the right solution is to have something like SchemaDStream, where the
schema of all the schemaRDD generated by it can be stored. Something I
really would like to see happen in the future :)
TD
On Thu, Jul 10, 2014 at 6:37 PM, Tobias Pfeiffer wrote:
> Hi,
>
> I think it would be great if
Hi,
I think it would be great if we could do the string parsing only once and
then just apply the transformation for each interval (reducing the
processing overhead for short intervals).
Also, one issue with the approach above is that transform() has the
following signature:
def transform(tran
Yes, this is what I tried, but thanks!
On Wed, Jul 9, 2014 at 6:02 PM, Tobias Pfeiffer wrote:
> Siyuan,
>
> I do it like this:
>
> // get data from Kafka
> val ssc = new StreamingContext(...)
> val kvPairs = KafkaUtils.createStream(...)
> // we need to wrap the data in a case class for register
Siyuan,
I do it like this:
// get data from Kafka
val ssc = new StreamingContext(...)
val kvPairs = KafkaUtils.createStream(...)
// we need to wrap the data in a case class for registerAsTable() to succeed
val lines = kvPairs.map(_._2).map(s => StringWrapper(s))
val result = lines.transform((rdd,
Hi guys,
I'm a new user to spark. I would like to know is there an example of how to
user spark SQL and spark streaming together? My use case is I want to do
some SQL on the input stream from kafka.
Thanks!
Best,
Siyuan
