chris-twiner commented on PR #48477: URL: https://github.com/apache/spark/pull/48477#issuecomment-2582255465
> > extra black box indirection that will not allow optimisation (e.g. constant folding etc.) > > I am not sure I am following here. Is the black box you talking about the "code hiding" inside the `Codec` type? I am not following why that would not be possible for Spark to do constant folding by just executing that code? Spark can only optimise Expressions, not general jvm byte code. Similarly, as Codec's are not an Expression, reference resolving will not take place on anything inside of that codec black box (stopping general use of ExpressionEncoders within the codec) - TransformingEncoder's are *very* good for the cases already implemented but fall short on cases where Spark could (as is currently done) optimise Expressions (including taking part in codegen). The use of custom TransformingEncoder codecs would also prohibit Dababricks Photon usage (or Fabrics upcoming equivalent) etc. as well in the general case (only provided as an example, the same is true for any custom Expression as well, most of spark-scalapb and frameless _should_ be compatible as they are resolved to standard Spark Expressions - except udfs -, Quality custom expressions are definitely not however). I'm more than happy to chat further directly (outside of this PR to minimise noise) if you would like to, but this PR is aimed at supporting 100% of the previous functionality, not around edge or sub use-cases that long-term may be better handled through another (possibly new) AgnosticEncoder (also hopefully keeping techdebt lower for users of frameless, spark-scalapb and Quality). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
