Ben Hollis created SPARK-59595:
----------------------------------
Summary: Data source expression normalization scales poorly for
wide schemas
Key: SPARK-59595
URL: https://issues.apache.org/jira/browse/SPARK-59595
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Ben Hollis
DataSourceStrategy.normalizeExprs searches the relation output for every
attribute reference being normalized. Its runtime is proportional to the number
of output attributes multiplied by the number of references. This causes
substantial planning overhead for wide schemas and projections. A local
benchmark with 10,000 attributes and 10,000 references took approximately 876
ms per call. The method may also be called several times per scan during
optimization and physical planning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
