Kontinuation commented on PR #1749: URL: https://github.com/apache/sedona/pull/1749#issuecomment-2576572653
Thank you for fixing this. I have just found that the way we are reading Parquet file metadata from footer is not optimal. We should simply use `ParquetFileReader.readFooter(configuration, path, SKIP_ROW_GROUPS).getFileMetaData` to read the file metadata, then we don't have to bother closing the readers. It also does less work by skip reading row groups, as row groups of huge files may be quite large. I don't think we need to add heavy weight tests for this by the way, as the fix is very straightforward, and it is not easy to write simple and quick tests for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
