Thanks TD. On Wed, Oct 25, 2017 at 6:42 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote:
> Please do not confuse old Spark Streaming (DStreams) with Structured > Streaming. Structured Streaming's offset and checkpoint management is far > more robust than DStreams. > Take a look at my talk - https://spark-summit.org/ > 2017/speakers/tathagata-das/ > > On Wed, Oct 25, 2017 at 9:29 PM, KhajaAsmath Mohammed < > mdkhajaasm...@gmail.com> wrote: > >> Thanks Subhash. >> >> Have you ever used zero data loss concept with streaming. I am bit >> worried to use streamig when it comes to data loss. >> >> https://blog.cloudera.com/blog/2017/06/offset-management- >> for-apache-kafka-with-apache-spark-streaming/ >> >> >> does structured streaming handles it internally? >> >> On Wed, Oct 25, 2017 at 3:10 PM, Subhash Sriram <subhash.sri...@gmail.com >> > wrote: >> >>> No problem! Take a look at this: >>> >>> http://spark.apache.org/docs/latest/structured-streaming-pro >>> gramming-guide.html#recovering-from-failures-with-checkpointing >>> >>> Thanks, >>> Subhash >>> >>> On Wed, Oct 25, 2017 at 4:08 PM, KhajaAsmath Mohammed < >>> mdkhajaasm...@gmail.com> wrote: >>> >>>> Hi Sriram, >>>> >>>> Thanks. This is what I was looking for. >>>> >>>> one question, where do we need to specify the checkpoint directory in >>>> case of structured streaming? >>>> >>>> Thanks, >>>> Asmath >>>> >>>> On Wed, Oct 25, 2017 at 2:52 PM, Subhash Sriram < >>>> subhash.sri...@gmail.com> wrote: >>>> >>>>> Hi Asmath, >>>>> >>>>> Here is an example of using structured streaming to read from Kafka: >>>>> >>>>> https://github.com/apache/spark/blob/master/examples/src/mai >>>>> n/scala/org/apache/spark/examples/sql/streaming/StructuredKa >>>>> fkaWordCount.scala >>>>> >>>>> In terms of parsing the JSON, there is a from_json function that you >>>>> can use. The following might help: >>>>> >>>>> https://databricks.com/blog/2017/02/23/working-complex-data- >>>>> formats-structured-streaming-apache-spark-2-1.html >>>>> >>>>> I hope this helps. >>>>> >>>>> Thanks, >>>>> Subhash >>>>> >>>>> On Wed, Oct 25, 2017 at 2:59 PM, KhajaAsmath Mohammed < >>>>> mdkhajaasm...@gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Could anyone provide suggestions on how to parse json data from kafka >>>>>> and load it back in hive. >>>>>> >>>>>> I have read about structured streaming but didn't find any examples. >>>>>> is there any best practise on how to read it and parse it with structured >>>>>> streaming for this use case? >>>>>> >>>>>> Thanks, >>>>>> Asmath >>>>>> >>>>> >>>>> >>>> >>> >> >
