Sean Zhong created SPARK-16898:
----------------------------------
Summary: Adds argument type information for typed logical plan
likMapElements, TypedFilter, and AppendColumn
Key: SPARK-16898
URL: https://issues.apache.org/jira/browse/SPARK-16898
Project: Spark
Issue Type: Bug
Reporter: Sean Zhong
Priority: Minor
Typed logical plan like MapElements, TypedFilter, and AppendColumn contains a
closure field: {{func: (T) => Boolean}}. For example class TypedFilter's
signature is:
{code}
case class TypedFilter(
func: AnyRef,
deserializer: Expression,
child: LogicalPlan) extends UnaryNode
{code}
>From the above class signature, we cannot easily find:
1. What is the input argument's type of the closure {{func}}? How do we know
which apply method to pick if there are multiple overloaded apply methods?
2. What is the input argument's schema?
With this info, it is easier for us to define some custom optimizer rule to
translate these typed logical plan to more efficient implementation, like the
closure optimization idea in SPARK-14083.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
