[
https://issues.apache.org/jira/browse/FLINK-28548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17568868#comment-17568868
]
Liu commented on FLINK-28548:
-----------------------------
[~zhangjing2000] [~lzljs3620320] Could you have a look? Thanks.
> The commit partition base path is not created when no data is sent which may
> cause FileNotFoundException
> --------------------------------------------------------------------------------------------------------
>
> Key: FLINK-28548
> URL: https://issues.apache.org/jira/browse/FLINK-28548
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.14.5, 1.15.1, 1.16.0
> Reporter: Liu
> Priority: Major
> Labels: pull-request-available
>
> The commit partition base path is not created when no data is sent which may
> cause FileNotFoundException. The exception is as following:
> {code:java}
> Caused by: java.io.FileNotFoundException: File
> /home/ljg/test_sql.db/flink_batch_test/.staging_1657697612169 does not exist.
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:771)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:120)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:828)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$20.doCall(DistributedFileSystem.java:824)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics$1.doCall(FileSystemLinkResolverWithStatistics.java:37)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at org.apache.hadoop.hdfs.perflog.PerfProxy.call(PerfProxy.java:49)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.hdfs.perflog.FileSystemLinkResolverWithStatistics.resolve(FileSystemLinkResolverWithStatistics.java:39)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:835)
> ~[hadoop-hdfs-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238)
> ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:238)
> ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.listStatus(ChRootedFileSystem.java:241)
> ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.listStatus(ViewFileSystem.java:376)
> ~[hadoop-common-2.6.0U203-cdh5.4.4.jar:?]
> at
> org.apache.flink.hive.shaded.fs.hdfs.HadoopFileSystem.listStatus(HadoopFileSystem.java:170)
> ~[flink-sql-connector-hive-2.3.6_2.12-1.15.0.jar:1.15.0]
> at
> org.apache.flink.connector.file.table.PartitionTempFileManager.listTaskTemporaryPaths(PartitionTempFileManager.java:87)
> ~[flink-connector-files-1.15.0.jar:1.15.0]
> at
> org.apache.flink.connector.file.table.FileSystemCommitter.commitPartitions(FileSystemCommitter.java:78)
> ~[flink-connector-files-1.15.0.jar:1.15.0]
> at
> org.apache.flink.connector.file.table.FileSystemOutputFormat.finalizeGlobal(FileSystemOutputFormat.java:89)
> ~[flink-connector-files-1.15.0.jar:1.15.0]
> at
> org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.finalizeOnMaster(InputOutputFormatVertex.java:153)
> ~[flink-dist-1.15.0.jar:1.15.0]
> at
> org.apache.flink.runtime.executiongraph.DefaultExecutionGraph.jobFinished(DefaultExecutionGraph.java:1190)
> ~[flink-dist-1.15.0.jar:1.15.0]
> ... 43 more {code}
> We should check whether the base path exists before listStatus for the path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
