chenkovsky commented on PR #14057: URL: https://github.com/apache/datafusion/pull/14057#issuecomment-2646781160
> > if a system/metadata column is not selected in project, after project, in spark system/metadata column's still a valid system/metadata column, but it's not that in #14362 > > Is this specific to Spark, or do most systems work this way? We should offer customizable options so users can choose their preference—unless this behavior is standard across most systems. I think both in this PR and #14362 are customizable, but it's harder in #14362, because many behaviors depends on the assumption "there is always a projection in the first pass". @jayzhan211 could you please also see the UT I put in #14362. you can see that this assumption is not correct in dataframe api. and you can also see the rowid save load problem. when I save a table to csv, it will also save rowid into csv. no system will do like this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
