Hi, After another day trying to get my head around WholeStageCodegenExec and InputAdapter and CollapseCodegenStages optimization rule I came to conclusion that it may have something to do with UnsafeRow vs GenericInternalRow/InternalRow so when a physical operator wants to _somehow_ participate in whole-stage codegen it can extend CodegenSupport trait and enable accessing GenericInternalRow by turning supportCodegen flag off.
I can understand how badly that can read, but without help from Spark SQL devs that's all I can figure out myself. Any help appreciated. Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Spark Structured Streaming https://bit.ly/spark-structured-streaming Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sun, Dec 10, 2017 at 10:34 PM, Stephen Boesch <java...@gmail.com> wrote: > A relevant observation: there was a closed/executed jira last year to > remove the option to disable the codegen flag (and unsafe flag as well): > https://issues.apache.org/jira/browse/SPARK-11644 > > 2017-12-10 13:16 GMT-08:00 Jacek Laskowski <ja...@japila.pl>: > >> Hi, >> >> I'm wondering why a physical operator like GenerateExec would >> extend CodegenSupport [1], but had the supportCodegen flag turned off? >> >> What's the meaning of such a combination -- be a CodegenSupport with >> supportCodegen off? >> >> [1] https://github.com/apache/spark/blob/master/sql/core/src >> /main/scala/org/apache/spark/sql/execution/GenerateExec.scala#L58-L64 >> >> [2] https://github.com/apache/spark/blob/master/sql/core/src >> /main/scala/org/apache/spark/sql/execution/GenerateExec.scala#L125 >> >> Pozdrawiam, >> Jacek Laskowski >> ---- >> https://about.me/JacekLaskowski >> Spark Structured Streaming https://bit.ly/spark-structured-streaming >> Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark >> Follow me at https://twitter.com/jaceklaskowski >> > >
