Cygwin on Wine: changing buffer content immediately after aync write() to a pipe corrupts written result
Dear list,
Recently we are investigating a Wine (Wine Staging) bug encountered by
Cygwin/MSYS2, which also suggest a potential bug in Cygwin.
When running xz on Cygwin/MSYS2 on Wine to decompress data and write
to a pipe, we found data corruption in the final result. This is
reported as Wine Staging bug 394:
https://bugs.wine-staging.com/show_bug.cgi?id=394
The command line to reproduce the bug is below:
$ cd /usr/share/gettext
$ xz -d -c < archive.dir.tar.xz | sha1sum # output result is random on Wine
Tested version:
Cygwin version
2.1.0, 2015/07/14 C:\cygwin\bin\cygwin1.dll
XZ version
$ xz --version
xz (XZ Utils) 5.2.1
liblzma 5.2.1
Wine Staging:
$ wine --version
wine-1.7.49-717-g3404140 (Staging)
In case anyone interesting, we also tested on msys2:
MSYS_NT-5.1 2.2.1(0.289/5/3) 2015-08-09 14:43 i686 Msys
***
With help from Lazhu (xz developer), Corrina and Sebastian, the
problem become a bit clearer. For archive, here are some testing
results we got:
1. This problem can't be reproduce on WinXP or Win7 according to my tests.
2. This problem doesn't happen with xz 5.0.8. Lazhu pointed out a
relate issue which doesn't happen in xz 5.0.8 but discovered by 5.2.1:
https://cygwin.com/ml/cygwin/2015-02/msg00575.html
That's very helpful information, finally we made sure the two problem
are different but get a clear idea what might be wrong.
3. Modified from Lazhu's previous test case, we wrote a new test case
to simulate the new problem, as attachment writer.c:
$ gcc writer.c -o writer
$ ./writer | sha1sum # the result is randomly on Wine but constant on Windows
The test case reproduce the problem on Wine, which generate randomly sha1sum.
If pipe is not used, then there is no data corruption:
$ ./write > good.txt && sha1sume good.txt # always good on Wine and Windows
However, the same test case doesn't reproduce anything wrong on WinXP
and Win7, sha1sum is always constant.
I tried modified buffer size, loop counts, etc inside the test case, I
also made sure EAGAIN is trigger on both Wine and Windows. I also
tried something like below to slow down reader speed:
$ ./write | pv -L512k | sha1sume
In any case, it's 100% reproduce on Wine, but never reproduce on Windows.
***
We are going to do some more test, will update this thread when we
have something interesting.
Thanks.
--
Regards,
Qian Hong
-
http://www.winehq.org
// gcc -std=gnu99 -Wall -Wextra writer.c -o writer
#include
#include
#include
#include
#include
#include
#include
#include
static void
fail(const char *str)
{
fprintf(stderr, "W: %s\n", str);
exit(EXIT_FAILURE);
}
int
main(void)
{
int flags = fcntl(STDOUT_FILENO, F_GETFL);
if (flags == -1)
fail("Cannot get stdout file status flags");
if (fcntl(STDOUT_FILENO, F_SETFL, flags | O_NONBLOCK) == -1)
fail("Setting stdout to non-blocking mode failed");
static unsigned char buf[64 << 10];
int blocking = 0;
int i = 0;
while (i < 256)
{
int ret = write(STDOUT_FILENO, buf, sizeof(buf));
if (ret == -1 && errno == EAGAIN)
{
blocking++;
continue;
}
if (ret != sizeof(buf))
fail("Unexpected number of bytes");
memset(buf, ++i, sizeof(buf));
}
if (!blocking)
fail("Was never blocking, read was too fast");
if (fcntl(STDOUT_FILENO, F_SETFL, flags) == -1)
fail("Restoring stdout file status flags failed");
if (close(STDOUT_FILENO))
fail("Error closing stdout");
return EXIT_SUCCESS;
}
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Cygwin on Wine: changing buffer content immediately after aync write() to a pipe corrupts written result
Hi Qian, On Aug 15 19:11, Qian Hong wrote: > Dear list, > > Recently we are investigating a Wine (Wine Staging) bug encountered by > Cygwin/MSYS2, which also suggest a potential bug in Cygwin. > [...] thanks for the detailed report and your efforts to reproduce the issue. As discussed on IRC, I'm going to use this for the release notes. The proposed Cygwin fix is in the 2.2.1-0.1 test release I'm uploading right now. Thanks, Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgp2c58sBaLwY.pgp Description: PGP signature
[ANNOUNCEMENT] TEST RELEASE: Cygwin 2.2.1-0.1
Hi Cygwin friends and users, I released a new TEST version of Cygwin. The version number is 2.2.1-0.1. This test release needs some good old-fashioned testing. 2.2.1 will be a bugfix release only: - Revert the following patch from 2.2.0: When started from a non-Cygwin process, check if $HOME starts with a slash (absolute POSIX path). Otherwise ignore it. - Fix output of /proc/cpuinfo in terms of cpu topology and cache size for modern CPUs and modern Windows OSes supporting more than 64 logical CPUs. - Don't try to perform RFC2307 owner/group mapping on Samba/NFS if account info is only fetched from local passwd/group files. Addresses: https://cygwin.com/ml/cygwin/2015-07/msg00270.html - Precautionally fix a potential data corruption problem in pipe I/O, only actually observered in Wine yet. However, MSDN language indicates this might be a problem on real Windows as well. Addresses: Freenode IRC #cygwin discussion, ML entry follows. Have fun, Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: commands spends time in cygheap_user
On Aug 14 22:20, Corinna Vinschen wrote: > On Aug 14 17:48, Corinna Vinschen wrote: > > On Aug 14 06:49, mku wrote: > > > I hope, not to follow a red herring again, but found some interesting. > > > > > > I did a fresh cygwin installation again but made an error not to copy my > > > old > > > .bashrc and .bashrc-profile files. > > > Within .bashrc I defined an alias for ls (ls='ls --color=auto > > > --show-control-chars'). > > > I noticed, that the time lag has went away. After a "ls --color=auto", the > > > time lag appeared again. > > > Comparing the strace output with/without the color flag, I found that the > > > time lag at "cygheap_user::ontherange" has gone and it now appeared at a > > > ldap_bind [ldap_init] that was called in the context of a symbolic link > > > to a > > > directory on a network share. The previous strace logs did not show a time > > > lag at this point, only at the cygheap_user entry. > > > > > > I deleted the "ln -s" entry and did not notice this big time lag any more > > > even with the color flag. > > > > > > I restored the "old" v2.2.0 version and cannot find the previous logged > > > time > > > lag. > > > As no files within the cygwin directory structure has been modified, it > > > seems, that some registry information has been "healed" by the multiple > > > fresh installations. > > > > > > PS: To do a fresh install I did a "backup" by renaming the original cygwin > > > folder. The "restore" was done by renaming the fresh installed cygwin > > > folder > > > and renaming the previous "backuped" folder back to cygwin. > > > > > > For me the issue is now closed. Thanks for your input. > > > > Thank you for this important point. That gives me an idea what > > happens and I guess this is something which needs fixing. These > > checks should also only happen if passwd/group in nsswitch.conf > > are set to "db". > > I just checked in a patch to address this issue, and I just created a > new developer snapshot for testing. > > Can you reproduce the situation by recreating the symlink and try again? > If you can reproduce it, can you give the latest Cygwin DLL from the > developer snapshot page https://cygwin.com/snapshots/ a try? The new 2.2.1-0.1 test release contains this fix as well. Please give it a try. Thanks, Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpgCyUv5kQel.pgp Description: PGP signature
Re: Cygwin on Wine: changing buffer content immediately after aync write() to a pipe corrupts written result
Hi Corinna, On Sat, Aug 15, 2015 at 7:27 PM, Corinna Vinschen wrote: > thanks for the detailed report and your efforts to reproduce the issue. > As discussed on IRC, I'm going to use this for the release notes. The > proposed Cygwin fix is in the 2.2.1-0.1 test release I'm uploading right > now. Thanks for the great help on irc. Also thanks for the test release. I tested 32bit version of the test release, on both Wine and Windows7, with both the xz command line and the test case writer.c attached in original post. I can confirm both Cygwin Wine and Cygwin Windows works now. $ sha1sum cygwin1-20150815.dll 929cd14b16bad6e00f4f51f61b9105756cfab8f6 cygwin1-20150815.dll $ file cygwin1-20150815.dll cygwin1-20150815.dll: PE32 executable (DLL) (console) Intel 80386 (stripped to external PDB), for MS Windows Will update if there is anything interesting. -- Regards, Qian Hong - http://www.winehq.org -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: run.exe fails to start XWin on Windows 8.1 through the shortcut
Hi Achim: Regarding run 1.3.3-1, if the command C:\cygwin64\bin\run.exe --quote /usr/bin/bash.exe -l -c "cd; /usr/bin/startxwin" is issued from the Windows command shell cmd, then C:\cygwin64\run.exe.stackdump is either not generated or not reproducibly generated (it happened just once in a long row of attempts). According to the Task Manager, the run process shortly appears and then disappears in one-two seconds. No other effects, except some disc activity, are visible. (In particular, XWin doesn't start.) \var\log\xwin remains untouched. Jaakov. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Shares with strange ACL settings
Corinna Vinschen writes: > Btw., can you please also check /proc/cpuinfo? The output is correct now for a SandyBridge dual-core CPU with logical processors (aka HT) and an IvyBridge dual-core CPU w/o HT. I've also tried to find out why the L3 cache is not reported for AMD in cpuinfo. It seems the reason is that it is logically part of the northbridge system, even if it is on-die and gets reported via CPUID. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: run.exe fails to start XWin on Windows 8.1 through the shortcut
Jaakov Jaakov writes: > Regarding run 1.3.3-1, Is there any reason why you're still bothering with 1.3.3 and not trying 1.3.4? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Shares with strange ACL settings
On 14/08/2015 15:45, Corinna Vinschen wrote: Marco, please see below in terms of L3 cache info. Thanks, [cut] Btw., can you please also check /proc/cpuinfo? As discussed, Cygwin's emulation fell short on L3 cache info. I now added code to fetch L3 cache info as well as correct processor topology information on Intel CPUs. For AMD CPUs the topology and cache info was already fine. Linux does not show L3 cache info for AMD CPUs afaics, so I also didn't add that to Cygwin. model name : Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz stepping: 9 cpu MHz : 2594.000 cache size : 3072 KB same L3 info as reported by $ hwloc-ls Machine (4383MB total) + NUMANode L#0 (P#0 4383MB) + Package L#0 + L3 L#0 (3072KB) L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 PU L#0 (P#0) PU L#1 (P#1) L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 PU L#2 (P#2) PU L#3 (P#3) Thanks, Corinna Regards Marco -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: run.exe fails to start XWin on Windows 8.1 through the shortcut
On 15/08/2015 17:25, Achim Gratz wrote: Jaakov Jaakov writes: Regarding run 1.3.3-1, Is there any reason why you're still bothering with 1.3.3 and not trying 1.3.4? Regards, Achim. May be as 1.3.4 is test and 1.3.3 is current ? Regards Marco -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: run.exe fails to start XWin on Windows 8.1 through the shortcut
Jaakov Jaakov writes: > Regarding run 1.3.3-1, Is there any reason why you're still bothering with 1.3.3 and not trying 1.3.4? Hi Achim and Jon: Several minites ago I've tried run 1.3.4-1. (Previously, I expected getting and using a test version to be much harder than it actually turned out to be.) And, voilà, starting XWin from the shortcut succeeded. Also starting from mintty and cmd. Said that: - huge, - tremendous, - enourmous, - exorbitant THANKS to you both, for finding a bug and correcting it. Best regards, Jaakov. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: Shares with strange ACL settings
On Aug 15 17:11, Achim Gratz wrote: > Corinna Vinschen writes: > > Btw., can you please also check /proc/cpuinfo? > > The output is correct now for a SandyBridge dual-core CPU with > logical processors (aka HT) and an IvyBridge dual-core CPU w/o HT. Thanks! > I've also tried to find out why the L3 cache is not reported for AMD in > cpuinfo. It seems the reason is that it is logically part of the > northbridge system, even if it is on-die and gets reported via CPUID. Ok, interesting. Still strange, isn't it? Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgp9UaPgWy0IB.pgp Description: PGP signature
Re: Shares with strange ACL settings
On Aug 15 17:41, Marco Atzeri wrote: > On 14/08/2015 15:45, Corinna Vinschen wrote: > >Marco, please see below in terms of L3 cache info. Thanks, > > > [cut] > > > >Btw., can you please also check /proc/cpuinfo? > > > >As discussed, Cygwin's emulation fell short on L3 cache info. I now > >added code to fetch L3 cache info as well as correct processor topology > >information on Intel CPUs. For AMD CPUs the topology and cache > >info was already fine. Linux does not show L3 cache info for AMD CPUs > >afaics, so I also didn't add that to Cygwin. > > > > model name : Intel(R) Core(TM) i5-3320M CPU @ 2.60GHz > stepping: 9 > cpu MHz : 2594.000 > cache size : 3072 KB > > same L3 info as reported by > > $ hwloc-ls > Machine (4383MB total) + NUMANode L#0 (P#0 4383MB) + Package L#0 + L3 L#0 > (3072KB) > L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 > PU L#0 (P#0) > PU L#1 (P#1) > L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 > PU L#2 (P#2) > PU L#3 (P#3) Nice, thanks for testing! Btw., I'm planning to add the sysconf cache output to 2.3.0, now that I know how to do it. Corinna -- Corinna Vinschen Please, send mails regarding Cygwin to Cygwin Maintainer cygwin AT cygwin DOT com Red Hat pgpQ164mKKcLf.pgp Description: PGP signature
Re: Shares with strange ACL settings
Corinna Vinschen writes: >> I've also tried to find out why the L3 cache is not reported for AMD in >> cpuinfo. It seems the reason is that it is logically part of the >> northbridge system, even if it is on-die and gets reported via CPUID. > > Ok, interesting. Still strange, isn't it? In any case, it's not due to an inability to get at the cache sizes. The code that fills those in has the sizes for all levels, but the cache size structure that gets used in /proc/cpuinfo is filled from L2 on AMD and L3 on Intel. If you need the full information on Linux you'd go to /sys/devices/system/cpu or use one of the more specialized programs that give you topology information anyway. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: run.exe fails to start XWin on Windows 8.1 through the shortcut
Jaakov Jaakov writes: > Several minites ago I've tried run 1.3.4-1. (Previously, I expected > getting and using a test version to be much harder than it actually > turned out to be.) OK, that explains it. > And, voilà, starting XWin from the shortcut > succeeded. Also starting from mintty and cmd. Said that: > - huge, > - tremendous, > - enourmous, - exorbitant > THANKS to you both, for finding a bug and correcting it. Well, thanks for reporting the bug with enough information to hunt it down, since I couldn't reproduce it at all. And of course to Jon for saving me the trouble of converting the stackdump back to something that makes sense and having a good eye on where the actual reason for the crash might be. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ DIY Stuff: http://Synth.Stromeko.net/DIY.html -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
Re: [PATCH] Change linker to gcc for cygwin
> On Aug 15, 2015, at 7:52 AM, Achim Gratz wrote: > > [Cygwin Perl maintainer here] > > Ivan Pozdeev writes: >> Currently, g++ is specified in hints/cygwin.sh >> It is quite a weighty install - approaches 100M with dependencies - >> and since it isn't really needed (see below), requiring it is >> completely unnecessary. >> Moreover, as a consequence, it's also required when building modules. > > I'm in general sympathetic with trying to keep dependencies small. yes, g++ is not required, only for -devel. g++ is not needed as perl run-time dep, only as build dep and devel dep, for people compiling XS modules. >> The current choice was introduced in commit >> 4f3b19ea9f1065e1d9d263b4c07fca1ba8f29276 >> as a replacement for `ld2'; related cygwin maillist message by Reini Urban: >> https://www.cygwin.com/ml/cygwin/2007-07/msg00665.html >> Neither the message not the linked perl5-porters thread contain any >> information on the choice of g++ over gcc. >> >> Maybe Reini himself can comment on his decision? g++ as ld was needed to be able to link C++ projects which also link to libperl, like gnome I think. > Maybe you could discuss things concerning Cygwin on the Cygwin mailing > list first before proposing changes for Cygwin upstream. I don't > pretend to know every detail of how and where Perl is used on Cygwin and > you shouldn't either. Maybe using g++ as a linker was something that > just happened or for reasons that are not valid anymore, but it's one of > the things I wouldn't want to change on a whim. It is still valid, Yaakov needed that. -- Problem reports: http://cygwin.com/problems.html FAQ: http://cygwin.com/faq/ Documentation: http://cygwin.com/docs.html Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple
