Omega359 commented on issue #14563: URL: https://github.com/apache/datafusion/issues/14563#issuecomment-2651322116
In looking into this issue I have a question for the db experts that happen to be following this issue. The with_column code builds a `Vec<Expr>` called fields in the dataframe and then projects all those fields in a new LogicalPlan that is returned in a new DataFrame. That's great ... except that projection seems to be doing far far too much work for fields that are completely unrelated to the `Expr` being passed into the `with_column(self, name, expr)` function. I am not an expert here by any means but I just can't see how preexisting expressions in the logical plan would need all the normalization/columnization that is performed since I would think that would have happened already. And if you are calling with_column hundreds of times like I do it definitely is doing that work over and over and over again. This essentially applies to any fn that call LogicalPlanBuilder.project such as `select`, `with_column`, `with_column_renamed`, My question is: could we instead have a way to tell the project that 'hey, these expressions are fine, trust me` and only do the work for the expression(s) that are new? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
