See below my queryExecution.optimizedPlan before apply my Rule.01 Project
[x#9, p#10, q#11, if (isnull(q#11)) null else UDF(q#11) AS udfB_10#28, if
(isnull(p#10)) null else UDF(p#10) AS udfA_99#93]02 +- InMemoryRelation
[x#9, p#10, q#11], true, 10000, StorageLevel(disk, memory, deserialized, 1
replicas)03 : +- *SerializeFromObject [assertnotnull(input[0, eic.R0,
true], top level non-flat input object).x AS x#9, unwrapoption(IntegerType,
assertnotnull(input[0, eic.R0, true], top level non-flat input object).p) AS
p#10, unwrapoption(IntegerType, assertnotnull(input[0, eic.R0, true], top
level non-flat input object).q) AS q#11]04 : +- *MapElements , obj#8:
eic.R005 : +- *DeserializeToObject newInstance(class
java.lang.Long), obj#7: java.lang.Long05 : +- *Range (0, 3,
step=1, splits=Some(2))In line 01 I need swap the position of udfA and udfB
this way:01 Project [x#9, p#10, q#11, if (isnull(p#10)) null else UDF(p#10)
AS udfA_99#93, if (isnull(q#11)) null else UDF(q#11) AS udfB_10#28]when I
try to change the order of the attributes in a Projection operation in
SparkSQL via Catalyst optimization the result of the query is modified to an
invalid value. Maybe I'm not doing everything is needed. I'm just changing
the order of NamedExpression objects in fields parameter:object
ReorderColumnsOnProjectOptimizationRule extends Rule[LogicalPlan] { def
apply(plan: LogicalPlan): LogicalPlan = plan resolveOperators { case
Project(fields: Seq[NamedExpression], child) => if
(checkCondition(fields)) Project(newFieldsObject(fields), child) else
Project(fields, child) case _ => plan } private def
newFieldsObject(fields: Seq[NamedExpression]): Seq[NamedExpression] = {
// compare UDFs computation cost and return the new NamedExpression list
. . . } private def checkCondition(fields: Seq[NamedExpression]): Boolean
= { // compare UDFs computation cost and return Boolean for decision off
change order on field list. . . . } . . .}Note: I'm adding my Rule on
extraOptimizations SparkSQL object:spark.experimental.extraOptimizations =
Seq(ReorderColumnsOnProjectOptimizationRule)Any suggestions will be of great
help.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
