Asif created SPARK-59114:
----------------------------
Summary: java.lang.ClassCastException: class
org.apache.spark.sql.catalyst.expressions.UnsafeRow cannot be cast to class
org.apache.spark.sql.vectorized.ColumnarBatch
Key: SPARK-59114
URL: https://issues.apache.org/jira/browse/SPARK-59114
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.3, 4.2.0, 4.3.0
Reporter: Asif
When a spark plan using a FileSourceScanExec as leaf, reading a parquet format
file, with default config (i.e parquet vector read enabled) is cached, the
SparkPlan created is something like
WholeStageCodegenExec -> ColumnToRowExec -> InputAdapter -> FileSourceScanExec.
The presence of ColumntoRowExec, implies that the child is going to output
ColumnarBatch as the RDD.
This SparkPlan is stored in the CacheManager, and since the CachedRDDBuilder's
buffers are lazy init, the spark plan is not executed at the time of caching.
Now after cached entry is created, and a query is executed involving the cached
plan, with SQLConf having
SQLConf.PARQUET_VECTORIZED_READER_ENABLED -> false,
then the cached plan is executed, with the this property, and as a result, the
FileSourceScanExec no longer returns RDD[ColumnarBatch] ( it is the RDD of Row
now).
While the CachedRDDBuilder code is expecting ColumnarBatch.
The fix proposed though is trivial, that is in the ColumnToRowExec.inputRDDs ,
explictly set the conf property of SQLConf.PARQUET_VECTORIZED_READER_ENABLED
as true.
The reasoning is that , since the plan contains ColumnToRowExec, that itself
implies that the child RDD should be returning a ColumnarBatch and not Row.
However , I am not sure if this is sufficient and may be
ColumnToRowExec.execute etc may also need to be tackled.
It is also not an elegant solution , more of work around.
The holistic solution should I suppose audit all the SQLConf properties which
otherwise can impact the Plan being used , such that it renders it totally
incompatible, May be such properties should be intialized in the Plan's
constructir itself at the time of creation and it should over-ride any existing
SQLConf.
I am exploring if its possible to store the nature of RDD return type ( Row or
Batch), as part of FileSourceScanExec constructor itself, so that its
unaffected by the subsequent change in conf.
A PR ( or may be two possible approaches fix will be filed shortly).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
