umehrot2 commented on issue #1764: URL: https://github.com/apache/hudi/issues/1764#issuecomment-650344638
@vinothchandar Agreed, I realized this soon after so didn't proceed with this approach. I am just thinking if we really need to wait for all files to appear here, or even if we need to wait, if at the end of the wait period the file is not present it should be safe to assume that file never got created. Atleast for S3 I can say that it is eventually consistent in the order of few 100 milliseconds. If after waiting so much (7 consistency checks by default) the file is not present, we can assume that it was never created. Another, approach can be that if parquet file didn't get created then we try to delete the marker file. But then again it can potentially fail to delete the marker as well. We can possibly do a combination of both. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
