IMHO - this is a bad idea esp in failure scenarios.
How about creating a subfolder each for the jobs?
On Sat, 17 Jul 2021 at 9:11 am, Eric Beabes
wrote:
> We've two (or more) jobs that write data into the same directory via a
> Dataframe.save method. We need to be able to figure out which job wrote
> which file. Maybe provide a 'prefix' to the file names. I was wondering if
> there's any 'option' that allows us to do this. Googling didn't come up
> with any solution so thought of asking the Spark experts on this mailing
> list.
>
> Thanks in advance.
>
--
Best Regards,
Ayan Guha
