Hi,
Add writeStream.option("quoteMode", "NONE")
By default Spark dataset api assumes that all the values MUST BE enclosed
in quote character (def: ") while writing to CSV files.
Akshay Bhardwaj
+91-97111-33849
On Tue, May 21, 2019 at 5:34 PM 杨浩 <[email protected]> wrote:
> We use struct streaming 2.2, when sinking as csv, a json str will automatic
> add "" for it, like an element is
>>
>> {"hello": "world"}
>
> result data in fs will be
>
>> "{\"hello\": \"world\"}"
>
>
> How to avoid the "",we only want
>
>> {"hello": "world"}
>
> code like
>
>> resultDS.
>> writeStream.
>> outputMode(OutputMode.Append()).
>> trigger(Trigger.ProcessingTime(TriggerInterval, TimeUnit.SECONDS)).
>> format("csv").
>> option("sep", "\t").
>> option("path", DIR).
>> option("checkpointLocation", CheckPointDir).
>> option("compression", "gzip").
>> partitionBy("event_day", "event_hour").
>> start.
>> awaitTermination()
>>
>
>
