[
https://issues.apache.org/jira/browse/HIVE-28963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17954220#comment-17954220
]
Ayush Saxena edited comment on HIVE-28963 at 5/27/25 5:52 AM:
--------------------------------------------------------------
ListFiles failing is a Hadoop issue, it is fixed 3.3.6+ & we are at 3.4.1 I
believe, So, I don't think this is an issue in our current code
was (Author: ayushtkn):
ListFiles failing is a Hadoop issue, it is fixed 3.3.6+ & we are at 3.4.1 I
believe, So, I don't think this is an issue in our code
> Handle FNF when ListFiles with recursive fails
> ----------------------------------------------
>
> Key: HIVE-28963
> URL: https://issues.apache.org/jira/browse/HIVE-28963
> Project: Hive
> Issue Type: Bug
> Reporter: Kokila N
> Assignee: Kokila N
> Priority: Major
>
> Insert Overwrite query with hive.acid.direct.insert.enabled=true writes data
> directly to the target directory(actual table location) in the MOVE stage.
> Say insert query is of writeid 5 and cleaner has got writeid 2 directories
> marked as obsolete
> {code:java}
> private static Path[] getDirectInsertDirectoryCandidatesRecursive(FileSystem
> fs,
> Path path, int skipLevels, PathFilter filter) throws IOException {
> String lastRelDir = null;
> HashSet<Path> results = new HashSet<Path>();
> String relRoot = Path.getPathWithoutSchemeAndAuthority(path).toString();
> if (!relRoot.endsWith(Path.SEPARATOR)) {
> relRoot += Path.SEPARATOR;
> }
> RemoteIterator<LocatedFileStatus> allFiles = fs.listFiles(path, true);
> while (allFiles.hasNext()) {
> LocatedFileStatus lfs = allFiles.next();
> .
> .
> }{code}
> _*fs.listFiles(path, true)*_
> - This recursively lists {*}all files{*}, even those that may be obsolete
> or deleted during iteration. So if Hive's cleaner deletes {{base_000002}}
> _after_ it's discovered by {{listFiles()}} but _before_ {{hasNext()}} tries
> to access it (for metadata resolution), we get a
> {{{}FileNotFoundException{}}}.
> *Cleaner:*
> There is no issue with the cleaner because it is deleting only the
> files/directories that are marked as obsolete.
> There is a fix for this issue in Hadoop upstream HADOOP-18662
> This was discovered when run hadoop version without this fix.
> Considering that we can handle this issue and also optimize from hive side.
> This scenario is race condition but also should occur rarely as cleaner
> should delete those directories in move stage of the insert between
> {{listFiles()and}} {{{}hasNext(){}}}to trigger this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
