sunchao commented on PR #4744: URL: https://github.com/apache/datafusion-comet/pull/4744#issuecomment-5745309809
Yes, strict masking in the native lambda path is the direction I had in mind. It addresses the evaluation semantics directly and keeps supported predicates native without maintaining a list of potentially dangerous expressions. A few details to preserve: - For `AND`, evaluate the RHS when the LHS is **true or null**. For `OR`, evaluate it when the LHS is **false or null**. This matters for SQL’s three-valued logic. - Skip RHS evaluation entirely when no elements need it, including zero-row inputs, and preserve element order so stateful expressions advance correctly. - Apply this recursively within lambda bodies and preserve it through expression rewrites. DataFusion’s `PhysicalExpr::evaluate_selection` looks worth reusing for filtering and scattering, with explicit handling for empty input. Please keep the existing empty-batch guard and speculative-serialization fallback. Runtime masking does not prevent exceptions during serialization. Before removing the Scala conditional guard entirely, we should also verify `IF`, `CASE WHEN`, and `COALESCE`. Could you add native-path regressions for the five reported cases, nullable boolean conditions, and nested predicates? Then rerun the benchmarks to quantify the masking overhead. That would give us a solid basis for reviewing the implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
