No problem! Take a look at this: http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-failures-with-checkpointing
Thanks, Subhash On Wed, Oct 25, 2017 at 4:08 PM, KhajaAsmath Mohammed < mdkhajaasm...@gmail.com> wrote: > Hi Sriram, > > Thanks. This is what I was looking for. > > one question, where do we need to specify the checkpoint directory in case > of structured streaming? > > Thanks, > Asmath > > On Wed, Oct 25, 2017 at 2:52 PM, Subhash Sriram <subhash.sri...@gmail.com> > wrote: > >> Hi Asmath, >> >> Here is an example of using structured streaming to read from Kafka: >> >> https://github.com/apache/spark/blob/master/examples/src/ >> main/scala/org/apache/spark/examples/sql/streaming/Structu >> redKafkaWordCount.scala >> >> In terms of parsing the JSON, there is a from_json function that you can >> use. The following might help: >> >> https://databricks.com/blog/2017/02/23/working-complex-data- >> formats-structured-streaming-apache-spark-2-1.html >> >> I hope this helps. >> >> Thanks, >> Subhash >> >> On Wed, Oct 25, 2017 at 2:59 PM, KhajaAsmath Mohammed < >> mdkhajaasm...@gmail.com> wrote: >> >>> Hi, >>> >>> Could anyone provide suggestions on how to parse json data from kafka >>> and load it back in hive. >>> >>> I have read about structured streaming but didn't find any examples. is >>> there any best practise on how to read it and parse it with structured >>> streaming for this use case? >>> >>> Thanks, >>> Asmath >>> >> >> >
