ls/stat on OneDrive causes download of files
If I invoke ls or anything else that does stat inside OneDrive folder it will trigger download of all files. OneDrive uses placeholder files[1] to represent remote files. I'm guessing reading file content in stat is to support detection of actually executable files as in here[2]? I think this should be disabled on non-hydrated placeholder files. Running `find` or 'ls -R` and having your entire OneDrive downloaded is extremely problematic. I could live without executable scripts in the OneDrive folder and it's easy to mark files as always offline to solve it. Another idea is to skip checking files with extensions known to be non-executable such as jpg (or just any extensions that is not known to be executable). This was previously reported in https://github.com/msys2/msys2-runtime/issues/206. [1] https://learn.microsoft.com/en-us/windows/win32/w8cookbook/placeholder-files [2] https://github.com/msys2/msys2-runtime/blob/msys2-3.4.10/winsup/cygwin/fhandler/disk_file.cc#L548 -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
On 2024-03-08 17:21, Corinna Vinschen wrote: On Mar 8 14:56, Corinna Vinschen via Cygwin wrote: On Mar 8 08:15, Jeffrey Altman via Cygwin wrote: On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: FILE_OPEN_NO_RECALL (0x0040) [...] This sounds like we could simply add this flag to all NtOpenFile used for path conversion or stat-like calls, without having to care for any file attributes specificially. Does that make sense? Sounds good, without even studying the other details... I speculate some more handling would still be needed to avoid executable detection via magic tags. Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but was not documented by Microsoft relatively recently. Thanks for the feedback, guys. Another suggestion would be to try opening the file with FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not required. See https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 Cygwin uses the minimum of required permissions in NtCreateFile/ NtOpenFile calls anyway. I'm just running a test cygwin DLL locally with a lot of added FILE_OPEN_NO_RECALL bits and a couple of added attribute checks for being offline to allow skipping some code. I think I'll push this change in a bit so we get a test release out so people using OneDrive can test. I pushed this change as well as a followup change to make sure we don't inadvertently recall an offline file. I also added handling for the Pinned and Unpinned attributes to chattr(1) and lsattr(1). The full set of changes can be tested by installing the Cygwin test release 3.6.0-0.77.g06aa5a751682. Please give it a try. If you encounter a situation which still results in recalling an offline file in a situation which doesn't qualify for it, please report. We will have to analyze that situation further then. Thanks, Corinna Thanks for doing this work so quickly. I'm not subscribed to this mailing list so I didn't see previous messages. I will try to check this in Cygwin this weekend but should tell you that I'm not cygwin user and now found a report of another user claiming this only happens in MSys and not in Cygwin. https://github.com/msys2/MSYS2-packages/issues/3049 -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: ls/stat on OneDrive causes download of files
I did more testing and found out that the problem does not happen in cygwin by default because cygwin mounts with acl which doesn't do header sniffing while msys uses noacl. Testing on an mp4 file in OneDrive, when I use noacl in cygwin it triggers the read as well. After upgrading to the test version the read is gone and an mp4 file is not executable. Thank you! On Fri, 8 Mar 2024 at 17:26, Marcin Wisnicki wrote: > > On 2024-03-08 17:21, Corinna Vinschen wrote: > > On Mar 8 14:56, Corinna Vinschen via Cygwin wrote: > >> On Mar 8 08:15, Jeffrey Altman via Cygwin wrote: > >>> On 3/8/2024 7:52 AM, Thomas Wolff via Cygwin wrote: > Am 08.03.2024 um 11:37 schrieb Corinna Vinschen via Cygwin: > > FILE_OPEN_NO_RECALL (0x0040) > > [...] > > This sounds like we could simply add this flag to all NtOpenFile > > used for path conversion or stat-like calls, without having to care > > for any file attributes specificially. > > > > Does that make sense? > Sounds good, without even studying the other details... > I speculate some more handling would still be needed to avoid executable > detection via magic tags. > > >>> Agreed. FILE_OPEN_NO_RECALL has been defined for at least a decade but > >>> was > >>> not documented by Microsoft relatively recently. > >> Thanks for the feedback, guys. > >> > >>> Another suggestion would be to try opening the file with > >>> FILE_READ_ATTRIBUTES instead of GENERIC_READ if the file data is not > >>> required. See > >>> > >>> https://github.com/microsoft/BuildXL/commit/4fb8e7ce07d243ccd95de0d66da551538a794493 > >> Cygwin uses the minimum of required permissions in NtCreateFile/ > >> NtOpenFile calls anyway. > >> > >> I'm just running a test cygwin DLL locally with a lot of added > >> FILE_OPEN_NO_RECALL bits and a couple of added attribute checks for > >> being offline to allow skipping some code. > >> > >> I think I'll push this change in a bit so we get a test release out > >> so people using OneDrive can test. > > I pushed this change as well as a followup change to make sure we don't > > inadvertently recall an offline file. I also added handling for the > > Pinned and Unpinned attributes to chattr(1) and lsattr(1). > > > > The full set of changes can be tested by installing the Cygwin test > > release 3.6.0-0.77.g06aa5a751682. > > > > Please give it a try. If you encounter a situation which still results > > in recalling an offline file in a situation which doesn't qualify for > > it, please report. We will have to analyze that situation further > > then. > > > > > > Thanks, > > Corinna > > Thanks for doing this work so quickly. I'm not subscribed to this > mailing list so I didn't see previous messages. > > I will try to check this in Cygwin this weekend but should tell you that > I'm not cygwin user and now found a report of another user claiming this > only happens in MSys and not in Cygwin. > > https://github.com/msys2/MSYS2-packages/issues/3049 > -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
