gatesn commented on issue #14993: URL: https://github.com/apache/datafusion/issues/14993#issuecomment-2703471110
> I can imagine situations where it wants to handle certain branches / conditions but would rather not re-implement others. Given these are DataFusion scalar expressions (rather than any relational algebra), can the implementation not just invoke the expression as a fallback? -- With Vortex, we've gone one step further and the scan accepts `projection: Expr, filter: Option<Expr>` where projection can arbitrarily select columns and apply scalar expressions to them. While this may be a little _too_ general for DataFusion, it works well provided the system has good support for struct types and expressions for manipulating them. We have select (to select/exclude fields from a struct), pack (to assemble fields into a new struct), getitem (to extract a field from a struct), and merge (to union multiple structs, although this can be implemented with getitem+pack). However this does have the downside of pushing disproportionate complexity onto the TableProvider for the simple case of projecting out a few columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
