sumitsingh-in commented on code in PR #20183:
URL: https://github.com/apache/hudi/pull/20183#discussion_r4181460298
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -346,10 +337,11 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
val dataStructTypeWithMandatoryPartitionFields =
StructType(dataStructType.fields ++ partitionSchema.fields.filter(f =>
mandatoryFields.contains(f.name) && !isNestedPartitionField(f.name)))
val dataSchema = HoodieSchemaUtils.pruneDataSchema(schema,
HoodieSchemaConversionUtils.convertStructTypeToHoodieSchema(dataStructTypeWithMandatoryPartitionFields,
sanitizedTableName), exclusionFields)
-
spark.sessionState.conf.setConfString("spark.sql.parquet.enableVectorizedReader",
supportVectorizedRead.toString)
Review Comment:
Removing the write is safe. Spark reads
`spark.sql.parquet.enableVectorizedReader` only to decide whether to convert
scan output to UnsafeRow, and it decides that before it calls
`buildReaderWithPartitionValues`, so the write never affected its own
statement. It only leaked `false` into later statements.
With the conf left at true, a row-based read such as a timestamp-millis
table on Spark 3.3/3.4 costs one extra row copy, same as stock Spark. The
failing case is the opposite one: the conf at false with vectorized rows, which
causes the ClassCastException this PR fixes.
Added a timestamp-millis test to `TestVectorizedReadDecisionPerScan` for COW
and MOR, covering narrow and wide scans with the conf at true. It passes on
Spark 3.3 and 4.2. Before this PR it fails only because the conf flips to
false, and the rows are still correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
