nsivabalan commented on code in PR #5141:
URL: https://github.com/apache/hudi/pull/5141#discussion_r846036990
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieBaseRelation.scala:
##########
@@ -333,7 +335,13 @@ object HoodieBaseRelation {
partitionedFile => {
val extension = FSUtils.getFileExtension(partitionedFile.filePath)
if (HoodieFileFormat.PARQUET.getFileExtension.equals(extension)) {
- parquetReader.apply(partitionedFile)
+ val iter = parquetReader.apply(partitionedFile)
+ if (iter.isInstanceOf[Closeable]) {
+ // register a callback to close parquetReader which will be executed
on task completion.
+ // when tasks finished, this method will be called, and release
resources.
+
Option(TaskContext.get()).foreach(_.addTaskCompletionListener[Unit](_ =>
iter.asInstanceOf[Closeable].close()))
Review Comment:
synced up offline. there is no follow up here. things are ok as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
