jayzhan211 commented on issue #14296: URL: https://github.com/apache/datafusion/issues/14296#issuecomment-2618344472
> > We have type coercion in logical plan now, consider the case where we want to separate logical types and physical types, should we add another type coercion layer in physical optimizer? > > I think it would make more sense to insist that the types line up exactly (have been resolved) in the Physical plans / physical types. But I haven't really thought through the implications of type coercion with a logical / physical type split I used to think type coercion for physical types (arrow::DataType) should be handled at the physical layer. However, if we consider it from another angle, the key distinction between the logical plan and the physical plan is that the logical plan has no knowledge of the actual data, only the metadata. At the logical level, we deal with abstractions like the Schema or scalar values, without interacting with the RecordBatch data itself. From this perspective, it makes more sense for type coercion to be part of the logical layer. Logical types can be seen as a more abstract or categorized version of physical types. A better way to describe this might be to think of them as "DataFusionType" and "ArrowType"—both of which are essentially logical concepts! In this context, I guess we don't need LogicalSchema from https://github.com/apache/datafusion/issues/12622 as well but interact with the arrow::Schema directly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
