Joris Van den Bossche created ARROW-8446:
--------------------------------------------
Summary: [Python][Dataset] Detect and use _metadata file in a list
of file paths
Key: ARROW-8446
URL: https://issues.apache.org/jira/browse/ARROW-8446
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Joris Van den Bossche
>From https://github.com/dask/dask/pull/6047#discussion_r402391318
When specifying a directory to {{ParquetDataset}}, we will detect if a
{{_metadata}} file is present in the directory and use that to populate the
{{metadata}} attribute (and not include this file in the list of "pieces",
since it does not include any data).
However, when passing a list of files to {{ParquetDataset}}, with one being
"_metadata", the metadata attribute is not populated, and the "_metadata" path
is included as one of the ParquetDatasetPiece objects instead (which leads to
an ArrowIOError during the read of that piece).
We _could_ detect it in a list of paths as well.
Note, I mentioned {{ParquetDataset}}, but if working on this, we should
probably directly do it in the datasets API-based version.
Also, I labeled this as Python and not C++ for now, as this might be something
that can be handled on the Python side (once the C++ side knows how to process
this kind of metadata -> ARROW-8062)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
