lifulong created SPARK-51692:
--------------------------------
Summary: can not find field for sql with partition prune and cte
Key: SPARK-51692
URL: https://issues.apache.org/jira/browse/SPARK-51692
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.5.5, 4.0.0
Environment: spark3.5
Reporter: lifulong
java.lang.IllegalStateException: Couldn't find client_id#49 in
[p_date#3543,client_id#3428,user_classify#3504]
at
org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:80)
at
org.apache.spark.sql.catalyst.expressions.BindReferences$$anonfun$bindReference$1.applyOrElse(BoundAttribute.scala:73)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
at
org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:76)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:461)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:437)
at
org.apache.spark.sql.catalyst.trees.TreeNode.transform(TreeNode.scala:405)
at
org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReference(BoundAttribute.scala:73)
at
org.apache.spark.sql.catalyst.expressions.BindReferences$.$anonfun$bindReferences$1(BoundAttribute.scala:94)
at scala.collection.immutable.List.map(List.scala:293)
at
org.apache.spark.sql.catalyst.expressions.BindReferences$.bindReferences(BoundAttribute.scala:94)
at
org.apache.spark.sql.execution.adaptive.PlanAdaptiveDynamicPruningFilters$$anonfun$apply$2.applyOrElse(PlanAdaptiveDynamicPruningFilters.scala:45)
at
org.apache.spark.sql.execution.adaptive.PlanAdaptiveDynamicPruningFilters$$anonfun$apply$2.applyOrElse(PlanAdaptiveDynamicPruningFilters.scala:40)
at
org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:461)
Rule PartitionPruning will add DynamicPruningSubquery Filter for partition
prune, DynamicPruningSubquery Node contains prune joinKeys and
filteringPlan, that info will finally used for gen
SubqueryAdaptiveBroadcastExec node
Rule ReplaceCTERefWithRepartition will gen new filed ref when replace CTE ref,
but do not replace buildKeys in SubqueryAdaptiveBroadcastExec node which
contains in CTE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
