Re: test -r or -x always return false on an NFS mount?
On Oct 13 21:00, Mario Emmenlauer wrote: > Hi Corinna, great to have you back :-) Thanks! > On 13.10.20 20:36, Corinna Vinschen wrote: > > > > > Everything seems to work quite well, and in `ls -la` I can see the > > > > > file permissions and user and group entries. But when using `test` > > > > > to check for read (`test -r`) or execute permissions (`test -x`), it > > > > > always returns false, even for readable files. `ls` on the other hand > > > > > shows the permissions correctly, and `cat`ing the files works without > > > > > problems. > > > > There's something fishy in your environment. NFS permissions from NFS > > shares mounted via Microsoft's NFSv3 are read using some internal > > function I got hinted at by the MSFT NFS guys at one point. This > > information is then exported as mode bits by Cygwin's stat(2) and used > > accordingly by Cygwin's access(2). > > > > Having said that, there's *no* magic at all in the user space > > applications other than using Cygwin's stat(2) and access(2) functions. > > > > Consequentially, using Cygwin's ls(1) or test(1) from coreutils, the > > results are the expected ones in both cases; just tried it myself, just > > to be sure. > > > > So, what's fishy? I don't know, but maybe you're using a non-Cygwin > > test(1) accidentally? > > I see your point, but to the best of my knowledge there is nothing > fishy. Its a freshly set up computer with an official Windows 10 x64 > from Microsoft directly. I've installed all latest updates including > the 2004 update. Cygwin was also just installed a few months ago. > > I did use a script to install Cygwin via its installer in an automated > fashion, but I've run the normal, graphical installer a few times since > then to make sure everything is nice and clean. > > Calling "which test" shows "/usr/bin/test", but since I use only > bash in all my scripts, I guess it anyways defaults to the builtin. Please check and try again with the stand-alone test(1), too. > The only "weak link" in my setup is the NFS mount. I'm no expert > in exporting NFS, nor in mounting NFS from Windows. Maybe something > is fishy there, albeit I did not use any special parameters or > quirks (again, to the best of my knowledge). What I can say is that > I'm limited to NFS v3 since its not a Server-Edition Windows. MSFT NFS is still limited to v3, anyway, on Windows Server, too. Be it as it is, this is no good reason for your results. I'm running AD as name service for the NFS client, but even if you have no name service for NFS installed, Cygwin fakes the uid/gid to match your own Windows user (just like on FAT). So you'd be bound to get more false positives, rather than false negatives. > Also, > I tried my best to open all NFS ports in the firewall but possibly > I'm not running one of the many extended NFS services like lockd > or something. I checked that most are installed, running and use a > static port, but its hard to be sure, since NFS seems to support > quite many "extras". The MSFT NFS client has two pre-configured entries in the Windows firewall called "Client for NFS (TCP-Out)" and "Client for NFS (UDP-Out)". These should be activated, nothing more should be required (unless using a name service, AD or RFC 2307, but that's a different story). > Is there anything I can do to isolate this further? Its a quite > curious case and I'm basically at the end of my wit... Actually, not really. It's weird in fact, given ls(1) shows the desired result. That would point to a bug in access(2), but there's no special code in access(2) for NFS. For filesystems not supporting ACLs (FAT, NFS, etc), it calls stat(2) and checks the st_mode bits against the requested access(2) mode based on the uid/gid of the caller, simple as that. Wrong uid/gid could wreck havoc, though. Maybe there is a bug in terms of using NFS without name service. Please call ls(1) and test(1) -r (not the bash builtin) on a file exposing the behaviour under strace like this: $ strace -o ls.trace /bin/ls -l $ strace -o test.trace /bin/test -r and send the trace files here, together with the output of the above ls(1) call and the output of id(1). Thanks, Corinna -- Corinna Vinschen Cygwin Maintainer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
On 14.10.20 10:28, Corinna Vinschen wrote: On Oct 13 21:00, Mario Emmenlauer wrote: On 13.10.20 20:36, Corinna Vinschen wrote: Everything seems to work quite well, and in `ls -la` I can see the file permissions and user and group entries. But when using `test` to check for read (`test -r`) or execute permissions (`test -x`), it always returns false, even for readable files. `ls` on the other hand shows the permissions correctly, and `cat`ing the files works without problems. There's something fishy in your environment. NFS permissions from NFS shares mounted via Microsoft's NFSv3 are read using some internal function I got hinted at by the MSFT NFS guys at one point. This information is then exported as mode bits by Cygwin's stat(2) and used accordingly by Cygwin's access(2). Having said that, there's *no* magic at all in the user space applications other than using Cygwin's stat(2) and access(2) functions. Consequentially, using Cygwin's ls(1) or test(1) from coreutils, the results are the expected ones in both cases; just tried it myself, just to be sure. So, what's fishy? I don't know, but maybe you're using a non-Cygwin test(1) accidentally? I see your point, but to the best of my knowledge there is nothing fishy. Its a freshly set up computer with an official Windows 10 x64 from Microsoft directly. I've installed all latest updates including the 2004 update. Cygwin was also just installed a few months ago. I did use a script to install Cygwin via its installer in an automated fashion, but I've run the normal, graphical installer a few times since then to make sure everything is nice and clean. Calling "which test" shows "/usr/bin/test", but since I use only bash in all my scripts, I guess it anyways defaults to the builtin. Please check and try again with the stand-alone test(1), too. Sorry, I should have mentioned this: it gives the same result. Is there anything I can do to isolate this further? Its a quite curious case and I'm basically at the end of my wit... Actually, not really. It's weird in fact, given ls(1) shows the desired result. That would point to a bug in access(2), but there's no special code in access(2) for NFS. For filesystems not supporting ACLs (FAT, NFS, etc), it calls stat(2) and checks the st_mode bits against the requested access(2) mode based on the uid/gid of the caller, simple as that. Hmm, now that you mention it, I just coincidentally found an issue with the `_stat` call in Microsoft Windows 2004 update. In the Apache thrift project I found that `_stat` stopped working on domain socket files on Windows. This sounds not directly identical, but could be well related. I did not try `_stat` in other situations, but something must have changed there very recently. The issue is reported upstream, but sadly for the wrong product (Visual Studio), so nobody is following up this report anymore: https://developercommunity.visualstudio.com/content/problem/1180994/-stat-fails-on-existing-domain-socket-file-when-bu.html (Scroll to the bottom to see the full report). Please call ls(1) and test(1) -r (not the bash builtin) on a file exposing the behaviour under strace like this: $ strace -o ls.trace /bin/ls -l $ strace -o test.trace /bin/test -r and send the trace files here, together with the output of the above ls(1) call and the output of id(1). I will to that! Thanks for your help and interest!! All the best, Mario Emmenlauer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
On Oct 14 11:06, Mario Emmenlauer wrote: > On 14.10.20 10:28, Corinna Vinschen wrote: > > Actually, not really. It's weird in fact, given ls(1) shows the > > desired result. That would point to a bug in access(2), but there's > > no special code in access(2) for NFS. For filesystems not supporting > > ACLs (FAT, NFS, etc), it calls stat(2) and checks the st_mode bits > > against the requested access(2) mode based on the uid/gid of the > > caller, simple as that. > > Hmm, now that you mention it, I just coincidentally found an issue > with the `_stat` call in Microsoft Windows 2004 update. In the Apache This is entirely unrelated. We're talking about Cygwin stat(2), not msvcrt.dll _stat(). Different source, different call. Corinna -- Corinna Vinschen Cygwin Maintainer -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Strange behaviour with winsymlinks:native
I've been doing some working around the problems with Cygwin 3.1.5+ WSL junction points in Docker and found three unexpected pieces of behaviour with CYGWIN=winsymlinks:native In all cases, these work as expected with the default symlink behaviour (i.e. CYGWIN unset or without a winsymlinks option). 1. Relative paths get unnecessarily resolved mkdir -p foo/bar cd foo/bar touch baz CYGWIN='winsymlinks:native' ln -s ../bar/baz link readlink link Result is baz, not ../bar/baz 2a. Links in the magic mount directories (/usr/bin and /usr/lib) get extra bits added (source is a default mount): cd /usr/lib CYGWIN='winsymlinks:native' ln -s ../share/terminfo terminfo2 readlink terminfo2 Result is ../usr/share/terminfo, which makes it valid from /lib (the "real" directory) but not in the virtual one 2b. Same but where target is a default mount cd /usr/libexec/p11-kit CYGWIN='winsymlinks:native' ln -s ../../bin/update-ca-trust trust-extract-compat2 readlink trust-extract-compat2 Result is ../../../bin/update-ca-trust, again it seems to be computing an extra level back to the "real" lib directory Are these behaviours expected? I can create those symbolic links manually with mklink with no problem, so I'm wondering if it's an outright bug or an unexpected consequence of something else. Thanks! David -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: test -r or -x always return false on an NFS mount?
On 14.10.20 13:50, Corinna Vinschen wrote: > On Oct 14 11:06, Mario Emmenlauer wrote: >> On 14.10.20 10:28, Corinna Vinschen wrote: >>> Actually, not really. It's weird in fact, given ls(1) shows the >>> desired result. That would point to a bug in access(2), but there's >>> no special code in access(2) for NFS. For filesystems not supporting >>> ACLs (FAT, NFS, etc), it calls stat(2) and checks the st_mode bits >>> against the requested access(2) mode based on the uid/gid of the >>> caller, simple as that. >> >> Hmm, now that you mention it, I just coincidentally found an issue >> with the `_stat` call in Microsoft Windows 2004 update. In the Apache > > This is entirely unrelated. We're talking about Cygwin stat(2), > not msvcrt.dll _stat(). Different source, different call. Yes, but Cygwin stat is implemented based on the Win32 posix layer too, or not? At least I got this impression from browsing the sources - albeit admittedly there are far too many indirections and ifdefs for me to really know what's going on... :-) :-) All the best, Mario -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: Unconsistent command-line parsing in case of UTF-8 quoted arguments
Thank you everyone, I now have a better understanding of how Windows
and Cygwin work (being rather a Linux guy, I was not really aware of
all of this).
However, there is still a question that is puzzling me. I now
understand _why_ things happen that way, but I am still wondering
whether this is really what we _want_. I mean, keeping the double
quotes around an UTF-8 argument just because it is not run from
Cygwin's bash sounds like a bug for me, doesn't it? (yet I definitely
understand the reasons that explain this behaviour). Since I cannot
run my program from bash, I have to resort to manually trimming the
quotes, which I would have liked to avoid.
I'd like to share a message that the maintainer of sshfs-win has
posted on Github [1], which is a follow-up to our discussions (he did
not know whether he was able to post in the mailing list without
subscribing first).
(besides, I unfortunately don't have much time currently to
investigate on this issue (for instance, I have not yet succeeded in
doing the same experiments with the very latest version of Cygwin), so
having his feedback is very valuable).
Here is what he says:
> It seems to me that the list is missing the important point
> about the double quote characters that should NOT be there
> regardless of how the é and ô characters are being interpreted.
> (As evidence of this: the Cygwin command line parser was able
> to break the command line into arguments correctly, but chose
> to retain the double quotes.)
>
> The choice of GetCommandLineA was for illustration purposes;
> had I used GetCommandLineW I would not be able to printf
> using %ls under CMD.EXE, because of code page issues. However
> here is a modified version of the test program that uses
> GetCommandLineW.
>
> #include
>
> wchar_t *GetCommandLineW(void);
>
> int main(int argc, char *argv[])
> {
> wchar_t *s = GetCommandLineW();
>
> for (wchar_t *p = s; *p; p++)
> printf("%04x %c%s",
> *p,
> 32 <= *p && *p < 127 ? *p : '.',
> (p - s) % 8 + 1 != 8 ? " " : "\n");
> printf("\n");
>
> for (int i = 0; argc > i; i++)
> printf("%d=%s\n", i, argv[i]);
>
> return 0;
> }
>
> I compiled this program under Cygwin to produce cyg.exe and ran
> it under Cygwin and CMD.EXE.
>
> Cygwin run:
> > billziss@xps:~/Projects/t$ locale
> LANG=en_US.UTF-8
> LC_CTYPE="en_US.UTF-8"
> LC_NUMERIC="en_US.UTF-8"
> LC_TIME="en_US.UTF-8"
> LC_COLLATE="en_US.UTF-8"
> LC_MONETARY="en_US.UTF-8"
> LC_MESSAGES="en_US.UTF-8"
> LC_ALL=
> billziss@xps:~/Projects/t$ ./cyg.exe "foo bar" "Domain\Jérôme"
> 0022 " 0043 C 003a : 005c \ 0055 U 0073 s 0065 e 0072 r
> 0073 s 005c \ 0062 b 0069 i 006c l 006c l 007a z 0069 i
> 0073 s 0073 s 005c \ 0050 P 0072 r 006f o 006a j 0065 e
> 0063 c 0074 t 0073 s 005c \ 0074 t 005c \ 0063 c 0079 y
> 0067 g 002e . 0065 e 0078 x 0065 e 0022 "
> 0=./cyg
> 1=foo bar
> 2=Domain\Jérôme
>
>
>
>
>
> CMD.EXE run:
>
> C:\Users\billziss\Projects\t>\Windows\System32\chcp.com
> Active code page: 437
>
> C:\Users\billziss\Projects\t>cyg.exe "foo bar" "Domain\Jérôme"
> 0063 c 0079 y 0067 g 002e . 0065 e 0078 x 0065 e 0020
> 0020 0022 " 0066 f 006f o 006f o 0020 0062 b 0061 a
> 0072 r 0022 " 0020 0022 " 0044 D 006f o 006d m 0061 a
> 0069 i 006e n 005c \ 004a J 00e9 . 0072 r 00f4 . 006d m
> 0065 e 0022 "
> 0=cyg
> 1=foo bar
> 2="Domain\Jérôme"
[1] https://github.com/billziss-gh/sshfs-win/pull/208
Thank you very much
Jérôme
Le mar. 13 oct. 2020 à 18:30, Kaz Kylheku (Cygwin)
<743-406-3...@kylheku.com> a écrit :
>
> On 2020-10-06 14:36, Jérôme Froissart wrote:
> > Here is an example C file
> > $ cat example.c
> > #include
> >
> > const char *GetCommandLineA(void);
> >
> > int main(int argc, char *argv[])
> > {
> > const char *s = GetCommandLineA();
> > printf("C=%s\n", s);
> >
> > for (int i = 0; argc > i; i++)
> > printf("%d=%s\n", i, argv[i]);
> >
> > return 0;
> > }
>
> Your program's comparison seems to be based on the
> hypothesis that Cygwin parses the GetCommandLineA() command line.
>
> But this hypothesis is almost certainly wrong.
>
> > Now, let's start a Windows shell (cmd.exe)
> > Note that I had to copy cygwin1.dll from my Cygwin installation
> > directory, otherwise binary.exe would not start.
> > I do not know whether there is a `locale` equivalent in Windows
> > command prompt, so I merely ran my program.
> > C:\Users\Public>binary.exe "foo bar" "Jérôme"
> > C=binary.exe "foo bar" "J□r□me"
> > 0=binary
> > 1=foo bar
> > 2="Jérôme"
>
> The "A" command line from GetCommandLineA has "tofu"
> characters: é and ô were not decoded properly.
>
> The é and ô characters we see in the Cygwin-parsed
> arguments coming into main could not have been recovered
>
Re: Unconsistent command-line parsing in case of UTF-8 quoted arguments
Le mer. 14 oct. 2020 à 23:47, Jérôme Froissart a écrit : > However, there is still a question that is puzzling me. I now > understand _why_ things happen that way, but I am still wondering > whether this is really what we _want_. I mean, keeping the double > quotes around an UTF-8 argument just because it is not run from > Cygwin's bash sounds like a bug for me, doesn't it? (yet I definitely > understand the reasons that explain this behaviour). Since I cannot > run my program from bash, I have to resort to manually trimming the > quotes, which I would have liked to avoid. Just to rephrase what is puzzling me: When I understood that sshfs-win had a bug when an argument contained diacritics, I expected many possible issues : mismatching codepages, poorly-handled encodings, implicit conversions between UTF-8 and Latin-1, etc., which would make some sense. But I definitely did not expect that "double quotes were not properly removed by the runtime", which (imho) does not make any sense. I hope I have managed to rephrase my problem clearly :D Thanks to all of you for your help! -- Problem reports: https://cygwin.com/problems.html FAQ: https://cygwin.com/faq/ Documentation:https://cygwin.com/docs.html Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
Re: UTF-8 quoted args passed to program include quotes when run from cmd
[changed subject]
On 2020-10-14 15:47, Jérôme Froissart wrote:
>> (As evidence of this: the Cygwin command line parser was able to break the
>> command line into arguments correctly, but chose to retain the double
>> quotes.)
>>
>> #include
>>
>> int main(int argc, char *argv[])
>> {
>> for (int i = 0; argc > i; i++)
>> printf("%d=%s\n", i, argv[i]);
>>
>> return 0;
>> }
>>
>> I compiled this program under Cygwin to produce cyg.exe and ran it under
>> Cygwin and CMD.EXE.
Please post compile and link command lines, as Cygwin can create native Windows
as well as its own Unix like executables, and the command line parsing may vary.
>> Cygwin run:
>>> billziss@xps:~/Projects/t$ locale
>> LANG=en_US.UTF-8
>> LC_CTYPE="en_US.UTF-8"
>> LC_NUMERIC="en_US.UTF-8"
>> LC_TIME="en_US.UTF-8"
>> LC_COLLATE="en_US.UTF-8"
>> LC_MONETARY="en_US.UTF-8"
>> LC_MESSAGES="en_US.UTF-8"
>> LC_ALL=
>> billziss@xps:~/Projects/t$ ./cyg.exe "foo bar" "Domain\Jérôme"
>> 0=./cyg
>> 1=foo bar
>> 2=Domain\Jérôme
>> CMD.EXE run:
>> C:\Users\billziss\Projects\t>cyg.exe "foo bar" "Domain\Jérôme"
>> 0=cyg
>> 1=foo bar
>> 2="Domain\Jérôme"
>>> Now, let's start a Windows shell (cmd.exe)
>>> Note that I had to copy cygwin1.dll from my Cygwin installation
>>> directory, otherwise binary.exe would not start.
>>> I do not know whether there is a `locale` equivalent in Windows
>>> command prompt, so I merely ran my program.
>>> C:\Users\Public>binary.exe "foo bar" "Jérôme"
>>> 0=binary
>>> 1=foo bar
>>> 2="Jérôme"
Your Windows CommandLineA/W outputs were confusing.
The point is that Cygwin programs run from cmd shell appear to receive UTF-8
arguments with the surrounding double quotes included intact, whereas the double
quotes are stripped when run from a Cygwin shell.
I think the charset needs verified by dumping each arg as hex bytes e.g.
//!/usr/bin/gcc -g -Og -Wall -Wextra -o quoted-arg-dump quoted-arg-dump.c
// quoted-arg-dump.c - dump quoted args under Cygwin and Windows shells
// outputs:
// $ ./quoted-arg-dump "foo bar" "Jérôme"
// 0 './quoted-arg-dump' 2e 2f 71 75 6f 74 65 64 2d 61 72 67 2d 64 75 6d 70
// 1 'foo bar' 66 6f 6f 20 62 61 72
// 2 'Jérôme' 4a c3 a9 72 c3 b4 6d 65
// >quoted-arg-dump "foo bar" "Jérôme"
// 0 'quoted-arg-dump' 71 75 6f 74 65 64 2d 61 72 67 2d 64 75 6d 70
// 1 'foo bar' 66 6f 6f 20 62 61 72
// 2 '"Jérôme"' 22 4a c3 a9 72 c3 b4 6d 65 22
// checks:
// $ grep -a '[éô]' unicode-symbols.txt
// é U+00E9 LATIN SMALL LETTER E WITH ACUTE
// ô U+00F4 LATIN SMALL LETTER O WITH CIRCUMFLEX
// $ grep -a '[éô]' unicode-symbols.txt | od -An -tx1z -w11
// c3 a9 20 20 55 2b 30 30 45 39 20 >.. U+00E9 <
// 20 4c 41 54 49 4e 20 53 4d 41 4c > LATIN SMAL<
// 4c 20 4c 45 54 54 45 52 20 45 20 >L LETTER E <
// 57 49 54 48 20 41 43 55 54 45 0a >WITH ACUTE.<
// c3 b4 20 20 55 2b 30 30 46 34 20 >.. U+00F4 <
// 20 4c 41 54 49 4e 20 53 4d 41 4c > LATIN SMAL<
// 4c 20 4c 45 54 54 45 52 20 4f 20 >L LETTER O <
// 57 49 54 48 20 43 49 52 43 55 4d >WITH CIRCUM<
// 46 4c 45 58 0a>FLEX.<
#include
int
main(int argc, char *argv[]) {
for (int a = 0; a < argc; ++a) {
printf("%d '%s'", a, argv[a]);
for (char *p = argv[a]; *p; ++p) {
printf(" %.2hhx", *p);
} // for chars
printf("\n");
} // for args
} // main()
This verifies that Cygwin does not strip double quotes from UTF-8 args when run
from Windows cmd, and the args are received and output as UTF-8 characters.
It might be interesting if you could also run from PowerShell and/or Terminal
for comparison to see if the Windows cmd behaviour is reproduced there.
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
[Data in binary units and prefixes, physical quantities in SI.]
--
Problem reports: https://cygwin.com/problems.html
FAQ: https://cygwin.com/faq/
Documentation:https://cygwin.com/docs.html
Unsubscribe info: https://cygwin.com/ml/#unsubscribe-simple
