Siyuan,
I do it like this:
// get data from Kafka
val ssc = new StreamingContext(...)
val kvPairs = KafkaUtils.createStream(...)
// we need to wrap the data in a case class for registerAsTable() to succeed
val lines = kvPairs.map(_._2).map(s => StringWrapper(s))
val result = lines.transform((rdd, time) => {
// execute statement
rdd.registerAsTable("data")
sqlc.sql(query)
})
Don't know if it is the best way, but it works.
Tobias
On Thu, Jul 10, 2014 at 4:21 AM, [email protected] <[email protected]> wrote:
> Hi guys,
>
> I'm a new user to spark. I would like to know is there an example of how
> to user spark SQL and spark streaming together? My use case is I want to do
> some SQL on the input stream from kafka.
> Thanks!
>
> Best,
> Siyuan
>
