Chao Sun created SPARK-59884:
--------------------------------
Summary: Avoid serializing cached columnar RDDs in whole-stage
closures
Key: SPARK-59884
URL: https://issues.apache.org/jira/browse/SPARK-59884
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
SparkPlan.executeColumnarRDD retains the LazyTry containing a plan's columnar
execution RDD, but unlike executeRDD it is not transient.
Whole-stage sort-merge join code generation adds the join plan to
CodegenContext.references for executor cleanup. If a descendant columnar plan
has initialized executeColumnarRDD, serializing the
WholeStageCodegenEvaluatorFactory also follows that driver-side cached RDD.
This unnecessarily captures the cached RDD object graph and can fail with
NotSerializableException when it contains driver-only state. Executable inputs
are already supplied to the evaluator separately.
A bounded reproducer builds a SortMergeJoinExec with a ColumnarToRowExec child
whose cached RDD has a nonserializable sentinel, obtains actual whole-stage
codegen references, initializes the columnar cache, and serializes the
evaluator with Spark's Java closure serializer. Upstream fails through:
WholeStageCodegenEvaluatorFactory.references -> SortMergeJoinExec ->
ColumnarToRowExec -> SparkPlan.executeColumnarRDD -> LazyTry.tryT -> cached RDD
-> nonserializable sentinel
Marking executeColumnarRDD @transient fixes this capture path and matches
executeRDD. Driver-side cache reuse is unchanged; serialization,
deserialization, and cleanup succeed with the fix.
The issue was reproduced using Apache Spark branch-4.2 source at
fc01a9e3165f9c371074390f4b3e6f7aeabef04b against public Spark 4.2.0
dependencies. The same nontransient field exists on current master and branches
4.x, 4.2, 4.1, and 4.0. Branch 3.5 directly invokes doExecuteColumnar and has
no such cache field.
This report concerns the demonstrated cached-RDD capture path. It does not
claim to eliminate every source of large task closures.
Proposed fix and regression test: https://github.com/apache/spark/pull/59146
Current-master validation at a026a0be4c88e7ace59d1da7c18ab73b0d81edde: the
focused regression passes with the fix and fails with NotSerializableException
when only the annotation is removed. The exact changed sources were compiled
against public Spark 4.2.0 dependencies; this is not a full current-master
build. Scoped Scalastyle passes with zero errors and warnings.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
