mbutrovich opened a new issue, #27134: URL: https://github.com/apache/datafusion/issues/27134
## Describe the bug `ListingTable` treats a file name of the form `key=value` as a partition directory. If a file stands in for a missing partition level, part of its file name becomes that partition's value. [`parse_partitions_for_path`](https://github.com/apache/datafusion/blob/b6b2e0e5d5f119f46b7fb6288827aee7041ab00d/datafusion/catalog-listing/src/helpers.rs#L453-L482) zips every path segment after the table path with the partition columns, and the file name is one of those segments. For `year=2024/month=02.parquet` under `PARTITIONED BY (year, month)`, the file name `month=02.parquet` matches column `month`, and the file is read with `month = '02.parquet'`. ## To Reproduce In `datafusion-cli` or a sqllogictest file: ```sql copy (values(3)) to 'scratch/t/year=2024/month=01/a.parquet'; create external table t (column1 bigint, year varchar, month varchar) stored as parquet partitioned by (year, month) location 'scratch/t/'; copy (values(4)) to 'scratch/t/year=2024/month=02.parquet'; -- the second file was written after CREATE, so skip the list files cache set datafusion.runtime.list_files_cache_limit = "0K"; select * from t order by column1; ``` This returns: ``` 3 2024 01 4 2024 02.parquet ``` ## Expected behavior The query returns only `3 2024 01`. `year=2024/month=02.parquet` is not under a `month=` directory, so it should be skipped like other files with an incomplete partition path. [`test_try_into_partitioned_file_partial_partition_skipped`](https://github.com/apache/datafusion/blob/b6b2e0e5d5f119f46b7fb6288827aee7041ab00d/datafusion/catalog-listing/src/helpers.rs#L772-L797) checks that `year=2024/data.parquet` is skipped for the same columns. Its comment says that happens because `data.parquet` doesn't match `month=...`, so the result depends on the file name. ## Additional context The partition check should cover only directory segments, which means dropping the last segment of the path before matching. apache/datafusion#26943 adds an opt-in strict mode that already rejects this file, because it counts directories without the file name. The default mode still reads it as above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
