asfimport opened a new issue, #28157:
URL: https://github.com/apache/arrow/issues/28157
Currently, the dataset writing (eg with `pyarrow.dataset.write_dataset`)
uses a fixed filename template (`"part\{i\}.ext"`). This means that when you
are writing to an existing dataset, you de facto overwrite previous data when
using this default template.
There is some discussion in ARROW-10695 about how the user can avoid this by
ensuring the file names are unique (the user can specify the
`basename_template` to be something unique). There is also ARROW-7706 about
silently doubling data (so _not_ overwriting existing data) with the legacy
`parquet.write_to_dataset` implementation.
It could be good to have a "mode" when writing datasets that controls the
different possible behaviours. And erroring when there is pre-existing data in
the target directory is maybe the safest default, because both appending vs
overwriting silently can be surprising behaviour depending on your expectations.
**Reporter**: [Joris Van den
Bossche](https://issues.apache.org/jira/browse/ARROW-12358) /
@jorisvandenbossche
#### Related issues:
- [[Python] [Dataset] Add partition_filename_cb to
ds.write_dataset()](https://github.com/apache/arrow/issues/28163) (fixes)
- [[Python] saving a dataframe to the same partitioned location silently
doubles the data](https://github.com/apache/arrow/issues/23949) (relates to)
- [[C++] [Dataset] Dataset repartition / filter /
update](https://github.com/apache/arrow/issues/28548) (is related to)
- [[C++][Dataset] Allow to use a UUID in the basename_template when writing
a dataset](https://github.com/apache/arrow/issues/26645) (is related to)
- [[C++] More fine-grained control of file creation in filesystem
layer](https://github.com/apache/arrow/issues/28274) (depends upon)
<sub>**Note**: *This issue was originally created as
[ARROW-12358](https://issues.apache.org/jira/browse/ARROW-12358). Please see
the [migration documentation](https://github.com/apache/arrow/issues/14542) for
further details.*</sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
