yihua commented on code in PR #20079:
URL: https://github.com/apache/hudi/pull/20079#discussion_r4170562125
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaEvolutionUtils.scala:
##########
@@ -80,9 +80,13 @@ class ParquetSchemaEvolutionUtils(sharedConf: Configuration,
protected var typeChangeInfos: java.util.Map[Integer, Pair[DataType,
DataType]] = null
- def getHadoopConfClone(footerFileMetaData: FileMetaData,
enableVectorizedReader: Boolean): Configuration = {
- // Clone new conf
- val hadoopAttemptConf = new Configuration(sharedConf)
+ /**
+ * Sets the file's requested read schema on the read configuration and
returns it for the task
+ * attempt context. The configuration belongs to this read only (see
[[SparkParquetReaderBase.read]]),
+ * so it is updated in place.
+ */
+ def getHadoopAttemptConf(footerFileMetaData: FileMetaData,
enableVectorizedReader: Boolean): Configuration = {
Review Comment:
nit: rename the method to a better name like `getHadoopConfWithSchema`?
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaEvolutionUtils.scala:
##########
@@ -80,9 +80,13 @@ class ParquetSchemaEvolutionUtils(sharedConf: Configuration,
protected var typeChangeInfos: java.util.Map[Integer, Pair[DataType,
DataType]] = null
- def getHadoopConfClone(footerFileMetaData: FileMetaData,
enableVectorizedReader: Boolean): Configuration = {
- // Clone new conf
- val hadoopAttemptConf = new Configuration(sharedConf)
+ /**
+ * Sets the file's requested read schema on the read configuration and
returns it for the task
+ * attempt context. The configuration belongs to this read only (see
[[SparkParquetReaderBase.read]]),
+ * so it is updated in place.
+ */
+ def getHadoopAttemptConf(footerFileMetaData: FileMetaData,
enableVectorizedReader: Boolean): Configuration = {
+ val hadoopAttemptConf = readConf
Review Comment:
We should revisit this after other improvements in #20064 are landed to make
sure there is no config sharing among multiple readers on the same executor due
to broadcasting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
