Ben Hollis created SPARK-59597:
----------------------------------
Summary: CTE predicate pushdown has quadratic planning cost for
wide CTEs
Key: SPARK-59597
URL: https://issues.apache.org/jira/browse/SPARK-59597
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Ben Hollis
`PushdownPredicatesAndPruneColumnsForCTEDef` restores attributes from CTE
references to CTE definitions by linearly searching the entire attribute
mapping for every referenced attribute. Planning time therefore grows
quadratically with CTE width.
The full rule takes about 100 microseconds for 100 columns, 1.65 milliseconds
for 500 columns, 6.53 milliseconds for 1,000 columns, and 26.54 milliseconds
for 2,000 columns. Wide CTEs with multiple references repeat this cost during
optimizer iterations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
