laid=6087618
From: Gera Shegalov
Sent: Wednesday, May 29, 2024 7:57:56 am
To: Prem Sahoo
Cc: eab...@163.com ; Vibhor Gupta ;
user @spark
Subject: Re: Re: EXT: Dual Write to HDFS and MinIO in faster way
I agree with the previous answers that (if requirements allow it
I agree with the previous answers that (if requirements allow it) it is
much easier to just orchestrate a copy either in the same app or sync
externally.
A long time ago and not for a Spark app we were solving a similar usecase
via
https://hadoop.apache.org/docs/r3.2.3/hadoop-project-dist/hadoop-h
I am looking for writer/comitter optimization which can make the spark
write faster.
On Tue, May 21, 2024 at 9:15 PM eab...@163.com wrote:
> Hi，
> I think you should write to HDFS then copy file (parquet or orc) from
> HDFS to MinIO.
>
> --
> eabour
>
>
> *From:*
Hi，
I think you should write to HDFS then copy file (parquet or orc) from HDFS
to MinIO.
eabour
From: Prem Sahoo
Date: 2024-05-22 00:38
To: Vibhor Gupta; user
Subject: Re: EXT: Dual Write to HDFS and MinIO in faster way
On Tue, May 21, 2024 at 6:58 AM Prem Sahoo wrote:
Hello Vibhor,
Th
