bmorck commented on issue #1092: URL: https://github.com/apache/datafusion-comet/issues/1092#issuecomment-2480110799
@viirya It's not clear to me that the issue is related to the issue on Project (19) however, I noticed that the appropriate `ColumnarToRow` operators are injected when the filter is able to be converted to native and hence the `EliminateRedundantTransitions` rule doesn't remove any nodes. I've seen this occur other queries as well. Here is another example: > +- AdaptiveSparkPlan (46) > +- == Current Plan == > Sort (26) > +- Project (25) > +- Filter (24) > +- Window (23) > +- Sort (22) > +- Exchange (21) > +- Union (20) > :- Project (10) > : +- Filter (9) > : +- Window (8) > : +- CometSort (7) > : +- ShuffleQueryStage (6) > : +- CometExchange (5) > : +- CometProject (4) > : +- CometFilter (3) > : +- CometSparkToColumnar (2) > : +- BatchScan (1) > +- Project (19) > +- Filter (18) > +- Window (17) > +- Sort (16) > +- ShuffleQueryStage (15) > +- Exchange (14) > +- Project (13) > +- Filter (12) > +- BatchScan (11) @huaxingao I didn't port over the PR yet, that's my next step and perhaps that resolves the issue. Will report here if I find that to be the case @andygrove Thanks for providing the new jar! Will try it out. We also are using an internal fork of comet (current changes are only enabling Spark 3.3 on some of the version restricted operators) so I will also rebase to include latest changes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
