[
https://issues.apache.org/jira/browse/HIVE-27447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735118#comment-17735118
]
Ayush Saxena commented on HIVE-27447:
-------------------------------------
Committed to master.
Thanx [~dkuzmenko] for the review!!!
> Iceberg: Queries failing due to FileSystem close errors due to column
> statistics
> --------------------------------------------------------------------------------
>
> Key: HIVE-27447
> URL: https://issues.apache.org/jira/browse/HIVE-27447
> Project: Hive
> Issue Type: Bug
> Reporter: Ayush Saxena
> Assignee: Ayush Saxena
> Priority: Major
> Labels: pull-request-available
>
> During fetching column statistics we are closing the filesystem in the HS2,
> FS is shared and cached, so closing it ain't a good idea.
> HS2 Logs show
> {noformat}
> 2023-06-16T16:57:09,515 DEBUG
> [45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22
> HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() by
> method:
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511));
> Key: (boroknagyz (auth:SIMPLE))@hdfs://localhost:20500; URI:
> hdfs://localhost:20500; Object Identity Hash: 14ec8672
> 2023-06-16T16:57:09,515 TRACE
> [45c1a38d-23c0-4e3a-96b7-c398bb215b2245c1a38d-23c0-4e3a-96b7-c398bb215b22
> HiveServer2-Handler-Pool: Thread-114] fs.FileSystem: FileSystem.close() full
> stack trace:
> java.lang.Throwable: null
> at
> org.apache.hadoop.fs.FileSystem.debugLogFileSystemClose(FileSystem.java:627)
> ~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
> at org.apache.hadoop.fs.FileSystem.close(FileSystem.java:2599)
> ~[hadoop-common-3.1.1.7.2.16.0-287.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1511)
> ~[hadoop-hdfs-client-3.1.1.7.2.16.0-287.jar:?]
> at
> org.apache.iceberg.mr.hive.HiveIcebergStorageHandler.canProvideColStatistics(HiveIcebergStorageHandler.java:444)
> ~[hive-iceberg-handler-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:1074)
> ~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]
> at
> org.apache.hadoop.hive.ql.stats.StatsUtils.collectStatistics(StatsUtils.java:298)
> ~[hive-exec-3.1.3000.2023.0.15.0-159.jar:3.1.3000.2023.0.15.0-159]{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
