m.queryName('test').format('console')\
.option('truncate', False).start()
q.awaitTermination()
From: Amit Joshi
Sent: Monday, 18 January 2021 20:22
To: Boris Litvak
Cc: spark-user
Subject: Re: [Spark Structured Streaming] Processing the data path coming from
kafka.
em into a
>> flat DF, ala
>> https://stackoverflow.com/questions/38243717/spark-explode-nested-json-with-array-in-scala
>>
>>
>>
>> Note that unless I am missing something you cannot access spark session
>> from foreach as code is not running on the
gt;
> Note that unless I am missing something you cannot access spark session
> from foreach as code is not running on the driver.
>
> Please say if it makes sense or did I miss anything.
>
>
>
> Boris
>
>
>
> *From:* Amit Joshi
> *Sent:* Monday, 18 January 202
Boris
From: Amit Joshi
Sent: Monday, 18 January 2021 17:10
To: Boris Litvak
Cc: spark-user
Subject: Re: [Spark Structured Streaming] Processing the data path coming from
kafka.
Hi Boris,
I need to do processing on the data present in the path.
That is the reason I am trying to make the dataframe.
> *Subject:* [Spark Structured Streaming] Processing the data path coming
> from kafka.
>
>
>
> Hi ,
>
>
>
> I have a use case where the file path of the json records stored in s3 are
> coming as a kafka
>
> message in kafka. I have to process the data us
Streaming] Processing the data path coming from
kafka.
Hi ,
I have a use case where the file path of the json records stored in s3 are
coming as a kafka
message in kafka. I have to process the data using spark structured streaming.
The design which I thought is as follows:
1. In kafka Spark structures
Hi ,
I have a use case where the file path of the json records stored in s3 are
coming as a kafka
message in kafka. I have to process the data using spark structured
streaming.
The design which I thought is as follows:
1. In kafka Spark structures streaming, read the message containing the
data p
