Re: [DISCUSS] Idempotent initialization of file systems

2023-05-16 Thread Moritz Mack
Thanks for your thoughts, Robert. On 15.05.23, 19:23, "Robert Bradshaw via dev" wrote: On Mon, May 15, 2023 at 8:38 AM Moritz Mack wrote: > > Hi all, > > I was just looking into an old issue again, SerializablePipelineOptions > calling FileSystems.setDefaultPipelineOptions on deserialization [1

Re: [DISCUSS] Idempotent initialization of file systems

2023-05-15 Thread Robert Bradshaw via dev
On Mon, May 15, 2023 at 8:38 AM Moritz Mack wrote: > > Hi all, > > I was just looking into an old issue again, SerializablePipelineOptions > calling FileSystems.setDefaultPipelineOptions on deserialization [1]. This > applies to various runners including Flink and Spark, but not Dataflow as far

[DISCUSS] Idempotent initialization of file systems

2023-05-15 Thread Moritz Mack
Hi all, I was just looking into an old issue again, SerializablePipelineOptions calling FileSystems.setDefaultPipelineOptions on deserialization [1]. This applies to various runners including Flink and Spark, but not Dataflow as far as I know. Problem: Current initialization of FileSystems thr