dwsmith1983 commented on issue #4224: URL: https://github.com/apache/datafusion-comet/issues/4224#issuecomment-5637096046
This landed in #4618 (7e413357e). `transform`, `exists`, `forall` and `aggregate` are `CometCodegenDispatch` serdes in spark/src/main/scala/org/apache/comet/serde/arrays.scala (`CometArrayTransform`, `CometArrayExists`, `CometArrayForAll`, `CometArrayAggregate`), so the lambda is evaluated by Spark's generated code over Arrow batches inside the Comet pipeline with no C2R/R2C transition, which is the outcome the JVM UDF bridge (#4170) was proposed for. `filter` is handled by `CometArrayFilter` in the same file: the `IsNotNull(x)` shape runs natively as `array_compact` and any other lambda goes through the same dispatcher. Suggest closing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
