yihua commented on code in PR #19914:
URL: https://github.com/apache/hudi/pull/19914#discussion_r4077752473
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/HoodieFileGroupReaderBasedFileFormat.scala:
##########
@@ -366,6 +387,12 @@ class HoodieFileGroupReaderBasedFileFormat(tablePath:
String,
.getSparkPartitionedFileUtils.getPathFromPartitionedFile(file))
fileSliceMapping.getSlice(fileGroupName) match {
case Some(fileSlice) if !isCount && (requiredSchema.nonEmpty ||
fileSlice.getLogFiles.findAny().isPresent) =>
+ if (!fileSlice.getLogFiles.findAny().isPresent && !isBootstrap
&& broadcastStockReader.isDefined) {
Review Comment:
Stepping back on the approach: I would rather we speed up the existing file
group reader path than add a second reader here, since the stock
`ParquetFileFormat` bypass has to re-implement every Hudi read semantic
(schema-on-read, type promotion, mandatory partition fields, vector columns,
required filters, ORC/multi-format) and the CI failures show how easy it is to
miss one. The `case _` branch below already skips `HoodieFileGroupReader` for a
base-only slice via `readBaseFile(file, baseFileReader.value, ...)` while
keeping the Hudi reader, so routing no-log COW slices there with the vectorized
`baseFileReader` should capture most of the win. Could you re-run the benchmark
through that path first, and only if a material gap remains, profile where it
is inside the Hudi reader and fix it there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
