Caizhi Weng created FLINK-25453:
-----------------------------------
Summary: LocalFileSystem#listStatus throws FileNotFoundException
when a file is deleted concurrently in the directory
Key: FLINK-25453
URL: https://issues.apache.org/jira/browse/FLINK-25453
Project: Flink
Issue Type: Bug
Components: API / Core
Affects Versions: 1.14.2
Reporter: Caizhi Weng
Add the following test case to {{LocalFileSystemTest}} to reproduce this issue.
{code:java}
@Test
public void myTest() throws Exception {
temporaryFolder.create();
Runnable r =
() -> {
try {
Path path = new Path(temporaryFolder.getRoot() + "/" +
UUID.randomUUID());
FileSystem fs = path.getFileSystem();
try (FSDataOutputStream out =
fs.create(path, FileSystem.WriteMode.NO_OVERWRITE))
{
OutputStreamWriter writer =
new OutputStreamWriter(out,
StandardCharsets.UTF_8);
writer.write("test");
writer.flush();
}
Thread.sleep(ThreadLocalRandom.current().nextInt(100));
fs.listStatus(new
Path(temporaryFolder.getRoot().toString()));
Thread.sleep(ThreadLocalRandom.current().nextInt(100));
fs.delete(path, false);
} catch (Exception e) {
e.printStackTrace();
}
};
List<Thread> threads = new ArrayList<>();
for (int i = 0; i < 100; i++) {
Thread thread = new Thread(r);
thread.start();
threads.add(thread);
}
for (Thread thread : threads) {
thread.join();
}
}
{code}
Exception stack
{code}
java.io.FileNotFoundException: File
/var/folders/y9/hqm_j18s105g5n8_xq00rd7c0000gp/T/junit8680341925762938456/f3b7f8a3-7092-464a-af7e-e7f8465c041d
does not exist or the user running Flink ('tsreaper') has insufficient
permissions to access it.
at
org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106)
at
org.apache.flink.core.fs.local.LocalFileSystem.listStatus(LocalFileSystem.java:167)
at
org.apache.flink.core.fs.local.LocalFileSystemTest.lambda$myTest$0(LocalFileSystemTest.java:90)
at java.lang.Thread.run(Thread.java:748)
{code}
This is because {{listStatus}} is not atomic. {{LocalFileSystem}} will first
get all file names in the directory and query for the status of each file. If
that file is removed after the start of {{listStatus}} but before the query
{{FileNotFoundException}} will be thrown.
Hadoop's {{RawLocalFileSystem}} handles this by ignoring
{{FileNotFoundException}}.
{code:java}
for (int i = 0; i < names.length; i++) {
try {
// Assemble the path using the Path 3 arg constructor to make sure
// paths with colon are properly resolved on Linux
results[j] = getFileStatus(new Path(f, new Path(null, null, names[i])));
j++;
} catch (FileNotFoundException e) {
// ignore the files not found since the dir list may have have changed
// since the names[] list was generated.
}
}
{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
