AdamGS commented on issue #20135: URL: https://github.com/apache/datafusion/issues/20135#issuecomment-4486393637
I'll try and clarify myself. My current implementation uses `try_pushdown_projection` to detect the `FileRowIndexFunc` scalar UDF and update the source's table schema. I've run into 2 cases: 1. `select file_row_index() from my_table where file_row_index() > 3`: This works well, `file_row_index` is in the `ProjectionExprs` that `try_pushdown_projection` and I can rewrite it there. 2. `select other_col from my_table where file_row_index > 3`: This doesn't work, as the UDF is not pushed down as part of the projection because its `Volatile` (regardless of its placement). The default implementation just returns `NULL` and the `FilterExec` node that's on top of the `DataSourceExec` ends up filtering everything away. The solution I've found is allowing UDFs that are `Volatile` and `ExpressionPlacement::should_push_to_leaves` to be pushed down (which they currently can't, this is a change in the `FilterPushdown` physical optimizer rule), so I can allow that pushdown to happen and I get to rewrite the UDF and the TableSchema with the virtual column. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
