I believe you need to set mapreduce.fileoutputcommitter.algorithm.version
to 2.
On Wed, Sep 19, 2018 at 10:45 AM Priya Ch
wrote:
> Hello Team,
>
> I am trying to write a DataSet as parquet file in Append mode partitioned
> by few columns. However since the job is time consuming, I would like to
Hello Team,
I am trying to write a DataSet as parquet file in Append mode partitioned
by few columns. However since the job is time consuming, I would like to
enable DirectFileOutputCommitter (i.e by-passing the writes to temporary
folder).
Version of the spark i am using is 2.3.1.
Can someone p
