xudong963 opened a new issue, #24210:
URL: https://github.com/apache/datafusion/issues/24210
### Describe the bug
When Hive partition columns are not kept in Parquet files,
`FileSinkConfig::output_schema()` still describes the sink input while the
writer schema removes those partition columns.
`ParquetFormat::create_writer_physical_plan` currently converts the input
ordering directly to Parquet `sorting_columns`, so the resulting column indices
can reference removed columns or the wrong positions in the physical file
schema.
Reading that metadata can then panic in `SchemaDescriptor::column` with an
out-of-bounds index.
### To Reproduce
Create and write a partitioned Parquet table whose ordering includes both a
partition column and regular columns, for example:
```sql
CREATE EXTERNAL TABLE sorted_partitioned_data (
a INT,
b VARCHAR,
part VARCHAR
)
STORED AS PARQUET
LOCATION '/tmp/sorted_partitioned_data'
PARTITIONED BY (part)
WITH ORDER (part ASC NULLS FIRST, a ASC NULLS FIRST, b DESC NULLS LAST);
INSERT INTO sorted_partitioned_data
VALUES (2, 'c', 'x'), (1, 'a', 'x'), (1, 'b', 'x');
```
The written Parquet file contains only `a` and `b`, but its
`sorting_columns` metadata is derived from the three-column input schema.
### Expected behavior
Parquet `sorting_columns` should omit ordering keys removed from the
physical file and remap retained keys to their indices in the writer schema.
The execution ordering used by `DataSinkExec` should remain unchanged.
### Additional context
The affected metadata-writing path was introduced by #19595. A downstream
fix and regression coverage were validated in
[massive-com/arrow-datafusion#71](https://github.com/massive-com/arrow-datafusion/pull/71).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
