[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file22281/11277.apple-fix-2.diff ___ Python tracker ___ ___ Python-bu

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: So sorry that i'm stressing this, hopefully it's the final message. Apples iterative kernel-update strategy resulted in these versions: 14:02 ~/tmp $ /usr/sbin/sysctl kern.version kern.version: Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 P

[issue11277] Crash with mmap and sparse files on Mac OS X

Ronald Oussoren added the comment: steffen: I have no idea what you are trying to say in your last message. Could you please try to stay on topic. -- ___ Python tracker ___ ___

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: @ Ronald Oussoren wrote: >if major <= 10: > # We're on OSX 10.6 or earlier > enableWorkaround() (You sound as if you participate in an interesting audiophonic event. 27" imac's are indeed great recording studio hardware. But no Coffee S

[issue11277] Crash with mmap and sparse files on Mac OS X

Ronald Oussoren added the comment: Steffen: _mac_ver_xml should not be dropped, it is a perfectly fine way to determine the system version. Discussing it is also off-topic for this issue, please keep the discussion focussed. Wrt. mailing Apple: I wouldn't expect and answer. Is there somethin

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file22273/11277.apple-fix.diff ___ Python tracker ___ ___ Python-bugs

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: Ok, this patch could be used. *Unless* the code is not protected by the GIL. - Gestalt usage is a bit more complicated according to http://www.cocoadev.com/index.pl?DeterminingOSVersion unless Python only supports OS X 10.4 and later. (And pla

[issue11277] Crash with mmap and sparse files on Mac OS X

Ned Deily added the comment: Victor, please do not use magic constants like that in C. The symbolic values are available in include files: #include SInt32 major = 0; SInt32 minor = 0; Gestalt(gestaltSystemVersionMajor, &major); Gestalt(gestaltSystemVersionMinor, &minor); if ((major == 10

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: Yes, you should check the Mac OS X version at runtime (as you should check the Linux kernel at runtime). platform.mac_ver() uses something like: sysv = _gestalt.gestalt('sysv') if sysv: major = (sysv & 0xFF00) >> 8 minor = (sysv & 0x00F0) >> 4 patch = (s

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: @ Ned Deily wrote (2011-06-07 19:43+0200): > Thanks for the update. Since the fix will be in a future > version of OS X 10.7 Lion, and which has not been released yet, > so it is not appropriate to change mmap until there has been an > opportunity to t

[issue11277] Crash with mmap and sparse files on Mac OS X

Ned Deily added the comment: Thanks for the update. Since the fix will be in a future version of OS X 10.7 Lion, and which has not been released yet, so it is not appropriate to change mmap until there has been an opportunity to test it. But even then, we would need to be careful about addi

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: Aehm, note that Apple has fixed the mmap(2) bug!! I'm still surprised and can't really believe it, but it's true! Just in case you're interested, i'll apply an updated patch. Maybe Ned Deily should have a look at the version check, which does not apply

[issue11277] Crash with mmap and sparse files on Mac OS X

Nadeem Vawda added the comment: sdaoden> @Nadeem: note that the committed versions of the tests would not sdaoden> show up the Mac OS X mmap() bug AFAIK, because there is an sdaoden> intermediate .close() of the file to be mmapped. Thanks for catching that. Should be fixed now. haypo> I now ag

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset 9b9f0de19684 by Nadeem Vawda in branch '2.7': Issue #11277: Fix tests - crash will not trigger if the file is closed and reopened. http://hg.python.org/cpython/rev/9b9f0de19684 New changeset b112c72f8c01 by Nadeem Vawda in branch '3.1': Issue #112

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: (Of course this may also be intentional, say. But then i would vote against it :), because it's better the tests bring out errors than end-user apps.) -- ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: @Nadeem: note that the committed versions of the tests would not show up the Mac OS X mmap() bug AFAIK, because there is an intermediate .close() of the file to be mmapped. The OS X bug is that the VMS/VFS interaction fails to provide a valid memory reg

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset 8d27d2b22394 by Nadeem Vawda in branch '2.7': Issue #11277: Add tests for mmap crash when using large sparse files on OS X. http://hg.python.org/cpython/rev/8d27d2b22394 -- ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset d5d4f2967879 by Nadeem Vawda in branch '3.1': Issue #11277: Add tests for mmap crash when using large sparse files on OS X. http://hg.python.org/cpython/rev/d5d4f2967879 New changeset e447a68742e7 by Nadeem Vawda in branch '3.2': Merge: #11277: Add

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset 201dcfc56e86 by Nadeem Vawda in branch '2.7': Issue #11277: Remove useless test from test_zlib. http://hg.python.org/cpython/rev/201dcfc56e86 -- ___ Python tracker __

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: > @haypo, @pitrou: Are there any objections to removing test_big_buffer() from Lib/test/test_zlib.py? I now agree Antoine: the test is useless. It can be removed today. About mmap: add a new test for this issue (mmap on Mac OS X and F_FULLSYNC) is a good ide

[issue11277] Crash with mmap and sparse files on Mac OS X

Nadeem Vawda added the comment: Thanks for the tests; I'll review and commit them tomorrow morning. > Even if that means that there is no test which checksums an > entire superlarge mmap() region. Bear in mind that the test is only to be removed from 2.7; it will still be present in the 3.* br

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file21885/11277-27.3.diff ___ Python tracker ___ ___ Python-bugs-list

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file21869/11277-27.2.diff ___ Python tracker ___ ___ Python-bugs-list

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: On Fri, 6 May 2011 02:54:07 +0200, Nadeem Vawda wrote: > I think so. [.] > it turns out that the OS X sparsefile crash is also covered by > LargeMmapTests.test_large_offset() in test_mmap [!!!]. [.] So i followed your suggestion and did not do somethin

[issue11277] Crash with mmap and sparse files on Mac OS X

Nadeem Vawda added the comment: haypo> Can we just remove the test? I think so. The test was originally intended to catch the case where crc32() or adler32() would get a buffer of >=4GB, and then silently truncate the size and produce an incorrect result (issue10276). However, 2.7's zlib doesn'

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: In fact i like my idea of using iterations. I have some time tomorrow, so if nobody complains until then, i write diffs for the tests of 3.x and 2.7 with these updates: - Two different target sizes: 1. 0x + x (7) 2. 0x7FFF + x (7) -

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: @haypo: trouble, trouble on the dev-list, as i've seen currently. Sorry, sorry. (Cannot subscribe, my DynIP's are often blacklisted ;) Of course my comments were completely wrong, as Ethan has pointed out correctly. This is all s**t. These are mmap(2)

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: @sdaoden(, @pitrou): Antoine proposes to skip the zlib "big buffer" (1 GB) test on 32 bits system. What do you think? On 64 bits system, we check a buffer of 2 GB-1 byte (0x7FFF bytes). Is the test useful or not? What do we test? Can you check if the tes

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: @haypo: Oh. Not: if sys.maxsize > _4G: # (64 bits system) crc32() and adler32() stores the buffer size into an # int, the maximum filesize is INT_MAX (0x7FFF) filesize = 0x7FFF crc_res = 0x709418e7 adle

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset e6a4deb84e47 by Victor Stinner in branch '2.7': Issue #11277: oops, fix checksum values of test_zlib on 32 bits http://hg.python.org/cpython/rev/e6a4deb84e47 -- ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset 7f3cab59ef3e by Victor Stinner in branch '2.7': Issue #11277: test_zlib tests a buffer of 1 GB on 32 bits http://hg.python.org/cpython/rev/7f3cab59ef3e -- ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: > error: [Errno 12] Cannot allocate memory @haypo: Well i told you i have no idea. These bots are 32 bit? I'll attach 11277-27.3.diff which does @skipUnless(not 32 bit). Note i'll test against >_4G - does this work (on 32 bit and in Python)? A pity t

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: "x86 debian parallel 2.7", "x86 Ubuntu Shared 2.7" and "x86 Tiger 2.7" fail with mmap.error('[Errno 12] Cannot allocate memory'). http://www.python.org/dev/buildbot/all/builders/x86%20Ubuntu%20Shared%202.7/builds/866/steps/test/logs/stdio http://www.python.org

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset 1ef2a7319849 by Victor Stinner in branch '2.7': Issue #11277: fix issue number in a test_zlib comment http://hg.python.org/cpython/rev/1ef2a7319849 -- ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: Reopen, test_zlib fails with Python 2.7 on Windows: == ERROR: test_big_buffer (test.test_zlib.ChecksumBigBufferTestCase) -- T

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: I commited mmap fix for Mac OS X, crc test on 2 GB file, and issue #8651 fix into Python 2.7. Use PY_SSIZE_T_CLEAN in zlibmodule.c is a new feature. I don't want to implement it, I don't need it, and I don't feel confortable in zlibmodule.c. Open a new issue

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset 618c3e971e80 by Victor Stinner in branch '2.7': (Merge 3.1) Issue #11277: mmap.mmap() calls fcntl(fd, F_FULLFSYNC) on Mac OS X http://hg.python.org/cpython/rev/618c3e971e80 -- ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file21855/11277-27.1.diff ___ Python tracker ___ ___ Python-bugs-list

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: > Should we fix Python 2.7? > - backport issue #8651 > - use PY_SSIZE_T_CLEAN in zlibmodule.c I really thought about this over night. I'm a C programmer and thus: - Produce no bugs - If you've produced a bug, fix it at once - If you've fixed a bug, sc

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: > @haypo: Python 2.7 is absolute horror. Oh, zlib doesn't use PY_SSIZE_T_CLEAN in Python 2.7. 11277-27.1.diff contains "# Issue #10276 - check that inputs >=4GB are handled correctly.". I don't understand this comment because the test uses a buffer of 2 GB +

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file21673/11277.zsum32.c ___ Python tracker ___ ___ Python-bugs-list

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: On Mon, 2 May 2011 01:22:41 +0200, STINNER Victor wrote: > @sdaoden: Can you try on Python 2.7? @haypo: Python 2.7 is absolute horror. But i tried and produced a (terrible - i don't know the test framework and that test_support stuff seems to have be

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: I am not able to check the fix, but the buildbots are :-) What should be done for Python 2.7? In Python 2.7, zlib.crc32() stores the buffer length into an int (so the maximum length is INT_MAX), and so test_zlib doesn't test a (sparse) file of 4 GB (ChecksumB

[issue11277] Crash with mmap and sparse files on Mac OS X

Roundup Robot added the comment: New changeset cb464f8fb3a1 by Victor Stinner in branch '3.1': Issue #11277: mmap calls fcntl(fd, F_FULLFSYNC) on Mac OS X to get around a http://hg.python.org/cpython/rev/cb464f8fb3a1 New changeset e9d298376dde by Victor Stinner in branch '3.2': (Merge 3.1) Issu

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file21717/11277.4.diff ___ Python tracker ___ ___ Python-bugs-list ma

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file21715/11277.3.diff ___ Python tracker ___ ___ Python-bugs-list ma

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: What do you think - i think this issue can really be closed now. I'll attach a final 11277.5.diff which has a less irritated and thus better understandable comment than .4.diff. I'll also drop .3 and .4. A lot of noise again 8| -- Added file: ht

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: > (My last reply-mail changed the title. Fixing.) Yeah, it's a common problem if you use the email interface :-/ -- ___ Python tracker ___ __

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: (My last reply-mail changed the title. Fixing.) -- title: test_zlib.test_big_buffer crashes under BSD (Mac OS X and FreeBSD) -> Crash with mmap and sparse files on Mac OS X ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

Steffen Daode Nurpmeso added the comment: Updated 11277.4.diff also includes mmap.rst update. (Maybe os.fsync() and os.sync() should be modified to really do that fcntl, too? I'll think i'll open an issue with patch soon.) -- Added file: http://bugs.python.org/file21717/11277.4.diff _

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: Oh, and can you add a comment explaining why F_FULLFSYNC is needed on Mac OS X in your patch? (If I understood correctly, it is needed to avoid crash with sparse files). -- ___ Python tracker

[issue11277] Crash with mmap and sparse files on Mac OS X

STINNER Victor added the comment: Oh, fcntl has already a F_FULLFSYNC constant, so we can use like fcntl.fcntl(fd, fcntl.F_FULLFSYNC) in Python. > can you add a sentence in mmap doc to explain that mmap.mmap() > does flush the file on Mac OS X and VMS? Hum, it does flush the file on VMS using

[issue11277] Crash with mmap and sparse files on Mac OS X

Changes by STINNER Victor : -- title: test_zlib.test_big_buffer crashes under BSD (Mac OS X and FreeBSD) -> Crash with mmap and sparse files on Mac OS X ___ Python tracker ___ _