when describing the resilience of the magic committer to failures during a
task commit, the docs state
"If the .pendingset file has been saved to the job attempt directory, the
task has effectively committed, it has just failed to report to the
controller. This will cause complications during job
promote the undocumented
"spark.hadoop.outputCommitCoordination.enabled" setting to a fully
supported option
thanks again
On Tue, Apr 23, 2024 at 8:49 AM Steve Loughran
wrote:
> On Sat, 20 Apr 2024 at 23:36, Dylan McClelland
> wrote:
>
> > when describing the resilience
