Ben Hollis created SPARK-59580:
----------------------------------
Summary: TreeNode construction allocates unused cache state
Key: SPARK-59580
URL: https://issues.apache.org/jira/browse/SPARK-59580
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Ben Hollis
Every Catalyst {{TreeNode}} eagerly allocates cache objects and initializer
closures for five properties. Most nodes never read all five properties, so
expression and plan construction pays for cache state that is never used.
This overhead compounds across large Catalyst trees. On the measured JVM,
constructing an otherwise unused {{Literal}} allocates 280 bytes, of which 199
bytes are avoidable cache overhead. The same overhead affects expressions,
logical plans, physical plans, and other {{TreeNode}} subclasses.
Acceptance criteria:
* Constructing tree nodes does not allocate cache objects for properties that
are never read.
* Existing {{TreeNode}} behavior and concurrency guarantees remain unchanged.
* A regression test detects renewed unused cache allocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
