Re: concurrent writes with dynamic partition overwrite mode

2019-09-05 Thread Steve Loughran
On Sun, Sep 1, 2019 at 7:54 PM Koert Kuipers wrote: > hi, > i am struggling to understand if concurrent writes to same basedir but > different partitions are save with file sources such as parquet. > > i tested this in spark 2.4 and spark 3.0.0-SNAPSHOT with real concurrent > jobs on hdfs and it

Re: concurrent writes with dynamic partition overwrite mode

2019-09-01 Thread Koert Kuipers
https://issues.apache.org/jira/browse/SPARK-28945 On Sun, Sep 1, 2019 at 2:53 PM Koert Kuipers wrote: > hi, > i am struggling to understand if concurrent writes to same basedir but > different partitions are save with file sources such as parquet. > > i tested this in spark 2.4 and spark 3.0.0-S

concurrent writes with dynamic partition overwrite mode

2019-09-01 Thread Koert Kuipers
hi, i am struggling to understand if concurrent writes to same basedir but different partitions are save with file sources such as parquet. i tested this in spark 2.4 and spark 3.0.0-SNAPSHOT with real concurrent jobs on hdfs and it seemed to work fine. admittedly this was a rather limited test. a