Hi Shuai,

Disclaimer: I'm not a spark guru, and what's written below are some
notes I took when reading spark source code, so I could be wrong, in
which case I'd appreciate a lot if someone could correct me.

(Yes, I did copy your disclaimer since it applies to me too. Sorry for
duplication :))

I'd say that the description is very well-written and clear. I'd only add that:

1. CodegenSupport allows custom implementations to optionally disable
codegen using supportCodegen predicate (that is enabled by default,
i.e. true)
2. CollapseCodegenStages is a Rule[SparkPlan], i.e. a transformation
of SparkPlan into another SparkPlan, that searches for sub-plans (aka
stages) that support codegen and collapse them together as a
WholeStageCodegen for which supportCodegen is enabled.
3. It is assumed that all Expression instances except CodegenFallback
support codegen.
4. CollapseCodegenStages uses the internal setting
spark.sql.codegen.maxFields (default: 200) to control the number of
fields in input and output schemas before deactivating whole-stage
codegen. See https://issues.apache.org/jira/browse/SPARK-14554.

NOTE: The magic number 200 (!) again. I asked about it few days ago
and in 
http://stackoverflow.com/questions/41359344/why-is-the-number-of-partitions-after-groupby-200

5. There are side-effecting logical commands that are executed for
their side-effects that are translated to ExecutedCommandExec in
BasicOperators strategy and won't take part in codegen.

Thanks for sharing your notes! Gonna merge yours with mine! Thanks.

Pozdrawiam,
Jacek Laskowski
----
https://medium.com/@jaceklaskowski/
Mastering Apache Spark 2.0 https://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Mon, Jan 2, 2017 at 6:30 PM, Shuai Lin <linshuai2...@gmail.com> wrote:
> Disclaimer: I'm not a spark guru, and what's written below are some notes I
> took when reading spark source code, so I could be wrong, in which case I'd
> appreciate a lot if someone could correct me.
>
>>
>> > Let me rephrase this. How does the SparkSQL engine call the codegen APIs
>> > to
>> do the job of producing RDDs?
>
>
> IIUC, physical operators like `ProjectExec` implements doProduce/doConsume
> to support codegen, and when whole-stage codegen is enabled, a subtree would
> be collapsed into a WholeStageCodegenExec wrapper tree, and the root node of
> the wrapper tree would call the doProduce/doConsume method of each operator
> to generate the java source code to be compiled into java byte code by
> janino.
>
> In contrast, when whole stage code gen is disabled (e.g. by passing "--conf
> spark.sql.codegen.wholeStage=false" to spark submit), the doExecute method
> of the physical operators are called so no code generation would happen.
>
> The producing of the RDDs is some post-order SparkPlan tree evaluation. The
> leaf node would be some data source: either some file-based
> HadoopFsRelation, or some external data sources like JdbcRelation, or
> in-memory LocalRelation created by "spark.range(100)". Above all, the leaf
> nodes could produce rows on their own. Then the evaluation goes in a bottom
> up manner, applying filter/limit/project etc. along the way. The generated
> code or the various doExecute method would be called, depending on whether
> codegen is enabled (the default) or not.
>
>> > What are those eval methods in Expressions for given there's already a
>> > doGenCode next to it?
>
>
> AFAIK the `eval` method of Expression is used to do static evaluation when
> the expression is foldable, e.g.:
>
>    select map('a', 1, 'b', 2, 'a', 3) as m
>
> Regards,
> Shuai
>
>
> On Wed, Dec 28, 2016 at 1:05 PM, dragonly <liyilon...@gmail.com> wrote:
>>
>> Thanks for your reply!
>>
>> Here's my *understanding*:
>> basic types that ScalaReflection understands are encoded into tungsten
>> binary format, while UDTs are encoded into GenericInternalRow, which
>> stores
>> the JVM objects in an Array[Any] under the hood, and thus lose those
>> memory
>> footprint efficiency and cpu cache efficiency stuff provided by tungsten
>> encoding.
>>
>> If the above is correct, then here are my *further questions*:
>> Are SparkPlan nodes (those ends with Exec) all codegenerated before
>> actually
>> running the toRdd logic? I know there are some non-codegenable nodes which
>> implement trait CodegenFallback, but there's also a doGenCode method in
>> the
>> trait, so the actual calling convention really puzzles me. And I've tried
>> to
>> trace those calling flow for a few days but found them scattered every
>> where. I cannot make a big graph of the method calling order even with the
>> help of IntelliJ.
>>
>> Let me rephrase this. How does the SparkSQL engine call the codegen APIs
>> to
>> do the job of producing RDDs? What are those eval methods in Expressions
>> for
>> given there's already a doGenCode next to it?
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-developers-list.1001551.n3.nabble.com/What-is-mainly-different-from-a-UDT-and-a-spark-internal-type-that-ExpressionEncoder-recognized-tp20370p20376.html
>> Sent from the Apache Spark Developers List mailing list archive at
>> Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>>
>

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Reply via email to