I am reading some data from kafka, and willing to save them to parquet on
hdfs with structured streaming.
The data from kafka is in JSON format. I try to convert them to
DataSet with spark.read.json(). However, I get the exception:
>
> Queries with streaming sources must be execute
start();
>>>>> query.awaitTermination();
>>>>>
>>>>>
>>>>> On Sat, Sep 9, 2017 at 4:20 PM, Felix Cheung <
>>>>> felixcheun...@hotmail.com> wrote:
>>>>>
>>>>>> What is newDS?
>>>>
string});
>>>> StreamingQuery query = ds.writeStream().start();
>>>> query.awaitTermination();
>>>>
>>>>
>>>> On Sat, Sep 9, 2017 at 4:20 PM, Felix Cheung >>> > wrote:
>>>>
>>>>> What is newDS?
>&
gt;>>> If it is a Streaming Dataset/DataFrame (since you have writeStream
>>>> there) then there seems to be an issue preventing toJSON to work.
>>>>
>>>> --
>>>> *From:* kant kodali
>>>> *Sent:* Sa
Streaming Dataset/DataFrame (since you have writeStream
>>> there) then there seems to be an issue preventing toJSON to work.
>>>
>>> ----------
>>> *From:* kant kodali
>>> *Sent:* Saturday, September 9, 2017 4:04:33 PM
>>> *To:* user @spark
>&
is newDS?
>> If it is a Streaming Dataset/DataFrame (since you have writeStream there)
>> then there seems to be an issue preventing toJSON to work.
>>
>> --
>> *From:* kant kodali
>> *Sent:* Saturday, September 9, 2017
k
> *Subject:* Queries with streaming sources must be executed with
> writeStream.start()
>
> Hi All,
>
> I have the following code and I am not sure what's wrong with it? I
> cannot call dataset.toJSON() (which returns a DataSet) ? I am using spark
> 2.2.0 so I a
sources must be executed with
writeStream.start()
Hi All,
I have the following code and I am not sure what's wrong with it? I cannot
call dataset.toJSON() (which returns a DataSet) ? I am using spark 2.2.0 so I
am wondering if there is any work around?
Dataset ds = newDS.toJSON().map(()-&
Hi All,
I have the following code and I am not sure what's wrong with it? I cannot
call dataset.toJSON() (which returns a DataSet) ? I am using spark 2.2.0 so
I am wondering if there is any work around?
Dataset ds = newDS.toJSON().map(()->{some function that
returns a string});
StreamingQuery
