Re: Possible Data Loss with RemoveOrphanFilesAction

2020-09-15 Thread Russell Spitzer
Yep, this is only a problem if you are running in an environment where your Paths may change due to authority or other parameters changing. Basically if any of the non "this is where the file is" information is mutable in your system and changes, you can have data loss with this bug. I'll write up

Re: Possible Data Loss with RemoveOrphanFilesAction

2020-09-14 Thread Ryan Blue
Thanks for the heads up on this. It sounds like this is not a concern for most people, but we should definitely add it to our maintenance docs to call it out in a warning. Would you like to open a PR for that? On Fri, Sep 11, 2020 at 3:45 PM Russell Spitzer wrote: > Because the RemoveOrphanFiles

Possible Data Loss with RemoveOrphanFilesAction

2020-09-11 Thread Russell Spitzer
Because the RemoveOrphanFilesAction uses Filesystem.list, the paths of files found in the file system can have an authority included in them based on the core-site.xml. This is determined when listing the files so the entries stored in the metadata tables do not necessarily have to match. URIs will