Technically, all GiotHub issue/PR discussions are on the mailing list because they get replicated to the git...@datafusion.apache.org <mailto:git...@datafusion.apache.org> mailing list, but it is definitely good to send emails to dev@ as well for issues like this that could affect downstream users. Thank you for starting this thread.
I am overall in favor of many of the goals of the proposal, but I also share the concern that it could have a significant impact on downstream projects, so we need to be careful not to remove functionality that others are depending on. In Comet, we would still want access to the physical types for optimization purposes, but would appreciate removing some of the current restrictions where an ExecutionPlan cannot mix and match physical types in its output, for example (we have input batches with plain Utf8 and other batches with dictionary-encoded Utf8 and currently have to add casts to ensure all batches use the same physical type, which adds unnecessary overhead). Thanks, Andy. > On Sep 24, 2024, at 6:46 AM, Filippo Rossi <filipporo...@hey.com.INVALID> > wrote: > > OT: this email address is being archived next year. This is my active > address reserved for OSS contributions: g...@filippo.dev (which I'm > adding as one of the receivers) > > Thanks, > Filippo > > On September 24, 2024, Piotr Findeisen <piotr.findei...@gmail.com> > wrote: >> Hi >> >> I am aware this message may seem redundant, given most communication >> happens in GitHub issues with some spill over to the ASF Slack. >> Posting this to make sure we follow "the apache way" (“If it didn’t >> happen on the mailing list, it didn’t happen.”). >> >> There is a great proposal from Filippo Rossi on the need to separate >> Logical types from physical representation >> <https://github.com/apache/datafusion/issues/11513> >> We hope this will go as far as removing arrow physical types >> completely from planning process, i.e. even physical plans will use >> the logical types. >> >> BTW I want the logical types to be "the types", and be the only thing >> function implementors need to be worried about. >> >> I posted also a proposal to separate AST and IR expression worlds: >> <https://github.com/apache/datafusion/issues/12604> >> >> Please take a look. >> >> Best >> Piotr >>
