allurefx opened a new issue, #18101:
URL: https://github.com/apache/datafusion/issues/18101
### Describe the bug
When using DataFrame.select() with aliased columns (e.g., col("Old
Name").alias("new_name")), the resulting DataFrame's schema doesn't reflect the
new column names when writing to parquet. This causes parquet writing to fail
with "field not found" errors.
### To Reproduce
1. Create a DataFrame from CSV with columns containing spaces/special
characters
2. Use [df.select(vec![col("Column Name").alias("column_name"),
...])](vscode-file://vscode-app/c:/Users/sekha/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html)
to sanitize column names
3. Attempt to write the DataFrame to parquet
4. Observe that parquet writing fails because it can't find the aliased
column names
### Expected behavior
Expected Behavior
The DataFrame schema should reflect the aliased column names after select(),
allowing parquet writing to succeed.
Actual Behavior
Parquet writing fails with "No field named [aliased_name]" errors, and the
schema still shows the original column names.
Workaround
Use [df.with_column_renamed("old_name",
"new_name")](vscode-file://vscode-app/c:/Users/sekha/AppData/Local/Programs/Microsoft%20VS%20Code/resources/app/out/vs/code/electron-browser/workbench/workbench.html)
instead of select() with aliases.
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
