[issue11935] MMDF/MBOX mailbox need utime
Steffen Daode Nurpmeso added the comment:
Let me close this!
I've just recently removed the real patch from my postman's "next"
branch, because even that real implementation doesn't work reliable.
I.e., please forget msg135791. It was true, but on the long run
mutt(1) sometimes sees all, sometimes only some (really nuts), but
most of the time it simply does not see just any box with new mail.
That is, that "plugged-in filesystem" is simply handled as a pendant.
Remarks: because that stdlib MBOX whispered
"Where Are Tho{u}, Brother"
to me all the {time}, i've done my own, also just recently:
== postman:
- test: 321 messages (5083760 bytes) [action=hunky-dory]
= Dispatched 321 tickets to 1 box.
[69853 refs] real 0m35.538s user 0m6.760s sys 0m0.904s
..
= Dispatched 1963 tickets to 1 box.
[93552 refs] real 0m38.860s user 0m8.697s sys 0m0.985s
== stdlib:
[83010 refs] real 1m3.862s user 0m10.151s sys 0m7.500s
[93217 refs] real 7m24.958s user 2m0.174s sys 1m35.163s
Was worth it.
Have a good time!
--
status: open -> closed
___
Python tracker
<http://bugs.python.org/issue11935>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11686] Update of some email/ __all__ lists
Steffen Daode Nurpmeso added the comment: Closing this... -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue11686> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11780] email.encoders are broken
Steffen Daode Nurpmeso added the comment: I think this is historic either? As far as i remember you solved it as part of another issue... -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue11780> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11701] email.parser.BytesParser().parse() closes file argument
Steffen Daode Nurpmeso added the comment: Right, and that is considered to be a non-issue due to that close() is allowed multiple times without causing harm. -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue11701> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11466] getpass.getpass doesn't close tty file
Changes by Steffen Daode Nurpmeso : -- nosy: -sdaoden ___ Python tracker <http://bugs.python.org/issue11466> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12102] mmap requires file to be synced
Steffen Daode Nurpmeso added the comment: > that doesn't make me any good Well - 'can only be better than myself, so i'll take that as yes :) -- ___ Python tracker <http://bugs.python.org/issue12102> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12102] mmap requires file to be synced
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file22020/12102.1.diff ___ Python tracker <http://bugs.python.org/issue12102> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11700] mailbox.py proxy updates
Steffen Daode Nurpmeso added the comment:
Hello, David, i'm about to remind you that this issue is still
open (and the next release is about to come soon).
I think we do agree at least in the fact that this is a bug :).
Well, your mailbox_close_twice.patch no. 2 still imports on the
current tip.
(I'm still a bit disappointed that you don't want to -a-r-m-
upgrade the proxies to the full implementation i've posted. But
it's ok. By the way: you're the first american i know who doesn't
want to upgrade his arms! And i do have had an ex-uncle who is
a fellow countryman of yours.)
Regards from Germany during kitschy pink sunset
--
Added file: http://bugs.python.org/file22095/11700.yeah-review.diff
___
Python tracker
<http://bugs.python.org/issue11700>
___diff --git a/Lib/mailbox.py b/Lib/mailbox.py
--- a/Lib/mailbox.py
+++ b/Lib/mailbox.py
@@ -1864,97 +1864,142 @@
"""Message with MMDF-specific properties."""
-class _ProxyFile:
-"""A read-only wrapper of a file."""
+class _ProxyFile(io.BufferedIOBase):
+"""A io.BufferedIOBase inheriting read-only wrapper for a seekable file.
+It supports __iter__() and the context-manager protocol.
+"""
+def __init__(self, file, pos=None):
+io.BufferedIOBase.__init__(self)
+self._file = file
+self._pos = file.tell() if pos is None else pos
+self._close = True
+self._is_open = True
-def __init__(self, f, pos=None):
-"""Initialize a _ProxyFile."""
-self._file = f
-if pos is None:
-self._pos = f.tell()
+def _set_noclose(self):
+"""Subclass hook - use to avoid closing internal file object."""
+self._close = False
+
+def _closed_check(self):
+"""Raise ValueError if not open."""
+if not self._is_open:
+raise ValueError('I/O operation on closed file')
+
+def close(self):
+if self._close:
+self._close = False
+self._file.close()
+del self._file
+self._is_open = False
+
+@property
+def closed(self):
+return not self._is_open
+
+def flush(self):
+# Not possible because it gets falsely called (issue 11700)
+#raise io.UnsupportedOperation('flush')
+pass
+
+def _read(self, size, read_method, readinto_arg=None):
+if size is None or size < 0:
+size = -1
+self._file.seek(self._pos)
+if not readinto_arg:
+result = read_method(size)
else:
-self._pos = pos
+result = read_method(readinto_arg)
+if result < len(readinto_arg):
+del readinto_arg[result:]
+self._pos = self._file.tell()
+return result
-def read(self, size=None):
-"""Read bytes."""
+def readable(self):
+self._closed_check()
+return True
+
+def read(self, size=-1):
+self._closed_check()
+if size is None or size < 0:
+return self.readall()
return self._read(size, self._file.read)
-def read1(self, size=None):
-"""Read bytes."""
+def read1(self, size=-1):
+self._closed_check()
+if size is None or size < 0:
+return b''
return self._read(size, self._file.read1)
-def readline(self, size=None):
-"""Read a line."""
+def readinto(self, by_arr):
+self._closed_check()
+return self._read(len(by_arr), self._file.readinto, by_arr)
+
+def readall(self):
+self._closed_check()
+self._file.seek(self._pos)
+if hasattr(self._file, 'readall'):
+result = self._file.readall()
+else:
+dl = []
+while 1:
+i = self._file.read(8192)
+if len(i) == 0:
+break
+dl.append(i)
+result = b''.join(dl)
+self._pos = self._file.tell()
+return result
+
+def readline(self, size=-1):
+self._closed_check()
return self._read(size, self._file.readline)
-def readlines(self, sizehint=None):
-"""Read multiple lines."""
+def readlines(self, sizehint=-1):
result = []
for line in self:
result.append(line)
-if sizehint is not None:
+if sizehint >= 0:
sizehint -= len(line)
if sizehint <= 0:
break
return result
+def seekab
[issue11700] mailbox.py proxy updates
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file22095/11700.yeah-review.diff ___ Python tracker <http://bugs.python.org/issue11700> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso added the comment: Aehm, note that Apple has fixed the mmap(2) bug!! I'm still surprised and can't really believe it, but it's true! Just in case you're interested, i'll apply an updated patch. Maybe Ned Deily should have a look at the version check, which does not apply yet, but i don't know any other way to perform exact version checking. (Using 10.6.7 is not enough, it must be 10.7.0; uname -a yet reports that all through, but no CPP symbol does!?) -- Added file: http://bugs.python.org/file22273/11277.apple-fix.diff ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso added the comment: @ Ned Deily wrote (2011-06-07 19:43+0200): > Thanks for the update. Since the fix will be in a future > version of OS X 10.7 Lion, and which has not been released yet, > so it is not appropriate to change mmap until there has been an > opportunity to test it. It's really working fine. That i see that day! (Not that they start to fix the CoreAudio crashes...) > But even then, we would need to be careful about adding > a compile-time test as OS X binaries are often built to be > compatible for a range of operating system version so avoid > adding compilation conditionals unless really necessary. > If after 10.7 is released and someone is able to test that it > works as expected, the standard way to support it would be to > use the Apple-supplied availability macros to test for the > minimum supported OS level of the build assuming it makes enough > of a performance difference to bother to do so Of course it only moves the delay from before mmap(2) to after close(2). Well, i don't know, if hardcoding is not an option, a dynamic sysctl(2) lookup may do: kern.version = Darwin Kernel Version 10.7.0: Sat Jan 29 15:17:16 PST 2011 This is obviously not the right one. :) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso added the comment:
Ok, this patch could be used.
*Unless* the code is not protected by the GIL.
- Gestalt usage is a bit more complicated according to
http://www.cocoadev.com/index.pl?DeterminingOSVersion
unless Python only supports OS X 10.4 and later.
(And platform.py correctly states that in _mac_ver_gestalt(),
but see below.)
- Due to usage of Gestalt, '-framework CoreServices' must be
linked against mmapmodule.c.
The Python configuration stuff is interesting for me, i managed
compilation by adding the line
mmap mmapmodule.c -framework CoreServices
to Modules/Setup, but i guess it's only OS X which is happy
about that.
platform.py: _mac_ver_xml() should be dropped entirely according
to one of Ned Deily's links ("never officially supported"), and
_mac_ver_gestalt() obviously never executed because i guess it
would fail due to "versioninfo". Unless i missed something.
By the way: where do you get the info from? "sys1", "sys2",
"sys3"? Cannot find it anywhere, only the long names, e.g.
gestaltSystemVersionXy.
Note that i've mailed Apple. I did not pay 99$ or even 249$, so
i don't know if there will be a response.
--
Ciao, Steffen
sdaoden(*)(gmail.com)
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
--
Added file: http://bugs.python.org/file22281/11277.apple-fix-2.diff
___
Python tracker
<http://bugs.python.org/issue11277>
___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst
--- a/Doc/library/mmap.rst
+++ b/Doc/library/mmap.rst
@@ -88,7 +88,8 @@
To ensure validity of the created memory mapping the file specified
by the descriptor *fileno* is internally automatically synchronized
- with physical backing store on Mac OS X and OpenVMS.
+ with physical backing store on operating systems where this is
+ necessary, e.g. OpenVMS, and some buggy versions of Mac OS X.
This example shows a simple way of using :class:`mmap`::
diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c
--- a/Modules/mmapmodule.c
+++ b/Modules/mmapmodule.c
@@ -25,6 +25,8 @@
#define UNIX
# ifdef __APPLE__
# include
+
+# include
# endif
#endif
@@ -65,6 +67,39 @@
#define my_getpagesize getpagesize
#endif
+# ifdef __APPLE__
+static void
+apple_osx_needs_fullsync(long *use_fullsync)
+{
+/* Issue #11277: mmap(2) bug with >32 bit sparse files.
+ * Apple fixed the bug before announcement of OS X "Lion", but since we
+ * need to handle buggy versions, perform a once-only check to see if the
+ * running kernel requires the expensive sync.
+ * >0: F_FULLSYNC is required, <0: kernel has mmap(2) bug fixed */
+SInt32 ver;
+
+*use_fullsync = 1;
+if (Gestalt(gestaltSystemVersion, &ver) != noErr && ver >= 0x1040) {
+if (Gestalt(gestaltSystemVersionMajor, &ver) != noErr)
+goto jleave;
+if (ver > 10) {
+*use_fullsync = -1;
+goto jleave;
+}
+
+if (Gestalt(gestaltSystemVersionMinor, &ver) != noErr)
+goto jleave;
+if (ver >= 7) {
+*use_fullsync = -1;
+goto jleave;
+}
+}
+
+jleave:
+return;
+}
+# endif /* __APPLE__ */
+
#endif /* UNIX */
#include
@@ -1128,8 +1163,14 @@
#ifdef __APPLE__
/* Issue #11277: fsync(2) is not enough on OS X - a special, OS X specific
fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */
-if (fd != -1)
-(void)fcntl(fd, F_FULLFSYNC);
+if (fd != -1) {
+/* (GIL protected) */
+static long use_fullsync /*= 0*/;
+if (!use_fullsync)
+apple_osx_needs_fullsync(&use_fullsync);
+if (use_fullsync > 0)
+(void)fcntl(fd, F_FULLFSYNC);
+}
#endif
#ifdef HAVE_FSTAT
# ifdef __VMS
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file22273/11277.apple-fix.diff ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso added the comment: @ Ronald Oussoren wrote: >if major <= 10: > # We're on OSX 10.6 or earlier > enableWorkaround() (You sound as if you participate in an interesting audiophonic event. 27" imac's are indeed great recording studio hardware. But no Coffee Shops in California - br.) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11728] mbox parser incorrect behaviour
Steffen Daode Nurpmeso added the comment:
Hello Valery Masiutsin, i recently stumbled over this while searching
for the link to the standart i've stored in another issue.
(Without being logged in, say.)
The de-facto standart (http://qmail.org/man/man5/mbox.html) says:
HOW A MESSAGE IS READ
A reader scans through an mbox file looking for From_ lines.
Any From_ line marks the beginning of a message. The reader
should not attempt to take advantage of the fact that every
From_ line (past the beginning of the file) is preceded by a
blank line.
This is however the recent version. The "mbox" manpage of my up-to-date
Mac OS X 10.6.7 does not state this, for example. It's from 2002.
However, all known MBOX standarts, i.e. MBOXO, MBOXRD, MBOXCL, require
proper quoting of non-From_ "From " lines (by preceeding with '>').
So your example should not fail in Python.
(But hey - are you sure *that* has been produced by Perl?)
You're right however that Python seems to only support the old MBOXO
way of un-escaping only plain "From " to/from ">From ", which is not
even mentioned anymore in the current standart - that only describes
MBOXRD ("(>*From )" -> ">"+match.group(1)).
(Lucky me: i own Mac OS X, otherwise i wouldn't even know.)
Thus you're in trouble if the unescaping is performed before the split..
This is another issue, though: "MBOX parser uses MBOXO algorithm".
;> - Ciao, Steffen
--
nosy: +sdaoden
___
Python tracker
<http://bugs.python.org/issue11728>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso added the comment: My suggestion to this would be that it should be outdated in the same way that Georg Brandl has suggested for changing the default encoding on python-dev [1], and add clear documentation on that, also in respect to the transition phase .. > The problem is that multiprocessing itself, by construction, > uses fork() with multiple threads. .. and maybe add some switches which allow usage of fork() for the time being. Today a '$ grep -Fir fork' does not show up threading.rst at all, which seems to be little in regard to the great problem. I would add a big fat note that multiprocessing.Process should be used instead today, because how about those of us who are not sophisticated enough to be appointed to standard committees? But anyway we should be lucky: fork(2) is UNIX specific, and thus it can be expected that all thread-safe libraries etc. are aware of the fact that they may be cloned by it. Except mine, of course. ,~) [1] http://mail.python.org/pipermail/python-dev/2011-June/112126.html -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso added the comment: > How do you think multiprocessing.Process launches a new process? But it's a single piece of code under control and even multi-OS/multi-architecture test-coverage, not a general purpose "Joe, you may just go that way and Python will handle it correctly"? What i mean is: ten years ago (or so), Java did not offer true selection on sockets (unless i'm mistaken) - servers needed a 1:1 mapping of threads:sockets to handle connections?! But then, a "this thread has finished the I/O, let's use it for something different" seems to be pretty obvious. This is ok if it's your professor who is forcefully misleading you into the wrong direction, but otherwise you will have problems, maybe sooner, maybe later (, maybe never). And currently there is not a single piece of documentation which points you onto the problems. (And there *are* really people without Google.) The problem is that it looks so simple and easy - but it's not. In my eyes it's an unsolvable problem. And for the sake of resource usage, simplicity and execution speed i appreciate all solutions which don't try to do the impossible. I want to add that all this does not really help just as long just *any* facility which is used by Python *itself* is not under control of atfork. Solaris e.g. uses atfork for it's memory allocator, because that is surely needed if anything else but async-safe facilities are used in the newly forked process. Can Python give that guarantee for all POSIX systems it supports? Good night. -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Steffen Daode Nurpmeso added the comment:
So sorry that i'm stressing this, hopefully it's the final message.
Apples iterative kernel-update strategy resulted in these versions:
14:02 ~/tmp $ /usr/sbin/sysctl kern.version
kern.version: Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011;
root:xnu-1504.15.3~1/RELEASE_I386
14:02 ~/tmp $ gcc -o zt osxversion.c -framework CoreServices
14:03 ~/tmp $ ./zt
OS X version: 10.6.8
apple_osx_needs_fullsync: -1
I.e. the new patch uses >10.7.0 or >=10.6.8 to avoid that
FULLFSYNC disaster (even slower than the Macrohard memory
allocator during "Wintel" partnership!), and we end up as:
14:03 ~/src/cpython $ ./python.exe -E -Wd -m test -r -w -uall test_mmap
Using random seed 8466468
[1/1] test_mmap
1 test OK.
P.S.: i still have no idea how to do '-framework CoreServices'
regulary. Like i've said in #11046 i never used GNU Autoconf/M4,
sorry. You know. Maybe the version check should be moved
somewhere else and simply be exported, even replacing the stuff
from platform.py? I don't know. Bye.
--
Ciao, Steffen
sdaoden(*)(gmail.com)
() ascii ribbon campaign - against html e-mail
/\ www.asciiribbon.org - against proprietary attachments
--
Added file: http://bugs.python.org/file22593/11277.apple-fix-3.diff
___
Python tracker
<http://bugs.python.org/issue11277>
___diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst
--- a/Doc/library/mmap.rst
+++ b/Doc/library/mmap.rst
@@ -88,7 +88,8 @@
To ensure validity of the created memory mapping the file specified
by the descriptor *fileno* is internally automatically synchronized
- with physical backing store on Mac OS X and OpenVMS.
+ with physical backing store on operating systems where this is
+ necessary, e.g. OpenVMS, and some buggy versions of Mac OS X.
This example shows a simple way of using :class:`mmap`::
diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c
--- a/Modules/mmapmodule.c
+++ b/Modules/mmapmodule.c
@@ -25,6 +25,8 @@
#define UNIX
# ifdef __APPLE__
# include
+
+# include
# endif
#endif
@@ -65,6 +67,44 @@
#define my_getpagesize getpagesize
#endif
+# ifdef __APPLE__
+static void
+apple_osx_needs_fullsync(long *use_fullsync)
+{
+/* Issue #11277: mmap(2) bug with >32 bit sparse files.
+ * Apple fixed the bug before announcement of OS X "Lion", but since we
+ * need to handle buggy versions, perform a once-only check to see if the
+ * running kernel requires the expensive sync. Fixed in 10.6.8, 10.7++.
+ * >0: F_FULLSYNC is required, <0: kernel has mmap(2) bug fixed */
+SInt32 ver;
+*use_fullsync = 1;
+
+if (Gestalt(gestaltSystemVersion, &ver) != noErr)
+goto jleave;
+/* SystemVersion(Major|Minor|BugFix) available at all? */
+if (ver < 0x1040)
+goto jleave;
+if (Gestalt(gestaltSystemVersionMajor, &ver) != noErr)
+goto jleave;
+if (ver > 10)
+goto jgood;
+if (Gestalt(gestaltSystemVersionMinor, &ver) != noErr)
+goto jleave;
+if (ver >= 7)
+goto jgood;
+if (ver < 6)
+goto jleave;
+if (Gestalt(gestaltSystemVersionBugFix, &ver) != noErr)
+goto jleave;
+if (ver < 8)
+goto jleave;
+jgood:
+*use_fullsync = -1;
+jleave:
+return;
+}
+# endif /* __APPLE__ */
+
#endif /* UNIX */
#include
@@ -1150,8 +1190,14 @@
#ifdef __APPLE__
/* Issue #11277: fsync(2) is not enough on OS X - a special, OS X specific
fcntl(2) is necessary to force DISKSYNC and get around mmap(2) bug */
-if (fd != -1)
-(void)fcntl(fd, F_FULLFSYNC);
+if (fd != -1) {
+/* (GIL protected) */
+static long use_fullsync /*= 0*/;
+if (!use_fullsync)
+apple_osx_needs_fullsync(&use_fullsync);
+if (use_fullsync > 0)
+(void)fcntl(fd, F_FULLFSYNC);
+}
#endif
#ifdef HAVE_FSTAT
# ifdef __VMS
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] Crash with mmap and sparse files on Mac OS X
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file22281/11277.apple-fix-2.diff ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso added the comment: Here is something unsorted and loose: - @neologix: One could argue that something had happened before the fsync(2), so that code which blindly did so is too dumb to do any right decision anyway. Even PEP 3151 won't help. - I favour haypos fullsync() approach, because that could probably make it into it. Yet Python doesn't offer any possibility to access NetBSD DISKSYNC stuff sofar. - Currently the programmer must be aware of any platform specific problems. I, for example, am not aware of Windows. How can i give any guarantee to users which (will) use my S-Postman on Windows? I need to put trust in turn into the Framework i am using - Python. And that makes me feel pretty breathless. - Fortunately Python is dynamic, so that one simply can replace os.fsync(). Works once only though (think signal handlers :=). + That is indeed the solution i'm using for my S-Postman, because *only* like this i can actually make Python's mailbox.py module reliable on Mac OS X! I can't give any guarantee for NetBSD, though i document it! + Aaarrg! I'm a liar!! I lie about - data integrity!!! --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue11877> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso added the comment: If Nir's analysis is right, and Antoines comment pushes me into this direction, (i personally have not looked at that code), then multiprocessing is completely brain-damaged and has been implemented by a moron. And yes, I know this is a bug tracker, and even that of Python. Nir should merge his last two messages into a single mail to python-dev, and those guys should give Nir or Thomas or a group of persons who have time and mental power a hg(1) repo clone and committer access to that and multiprocessing should be rewritten, maybe even from scratch, but i dunno. For the extremely unlikely case that all that doesn't happen maybe the patch of neologix should make it? --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso added the comment: Um, and just to add: i'm not watching out for anything, and it won't and it can't be me: ?0%0[steffen@sherwood sys]$ grep -F smp CHANGELOG.svn -B3 | grep -E '^r[[:digit:]]+' | tail -n 1 r162 | steffen | 2006-01-18 18:29:58 +0100 (Wed, 18 Jan 2006) | 35 lines --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso added the comment: P.S.: I have to apologize, it's Tomaž, not Thomas. (And unless i'm mistaken this is pronounced "TomAsch" rather than the english "Tommes", so i was just plain wrong.) --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue6721] Locks in python standard library should be sanitized on fork
Steffen Daode Nurpmeso added the comment: > > then multiprocessing is completely brain-damaged and has been > > implemented by a moron. > > Please do not use this kind of language. > Being disrespectful to other people hurts the discussion. So i apologize once again. 'Still i think this should go to python-dev in the mentioned case. (BTW: there are religions without "god", so whom shall e.g. i praise for the GIL?) --Steffen Ciao, sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments -- ___ Python tracker <http://bugs.python.org/issue6721> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso added the comment:
> > Even PEP 3151 won't help.
>
> I don't understand. If the syscall supposed to flush the disk's buffer
> cache fails - be it fcntl() or sync_file_range() - I think the error
> should be propagated, not silently ignored and replaced with another
> syscall which doesn't have the same semantic. That's all.
I'm with you theoretically - of course errors should be propagated
to users so that they can act accordingly.
And this is exactly what the patches do, *unless* it is an error
which is not produced by the native fsync(2) call:
-- >8 --
?0%0[steffen@sherwood tmp]$ cat t.c
#include
#include
#include
#include
#include
int main(void) {
int r = fcntl(2, F_FULLFSYNC);
fprintf(stderr, "1. %d: %d, %s\n", r, errno, strerror(errno));
errno = 0;
r = fsync(2);
fprintf(stderr, "2. %d: %d, %s\n", r, errno, strerror(errno));
return 0;
}
?0%0[steffen@sherwood tmp]$ gcc -o t t.c && ./t
1. -1: 25, Inappropriate ioctl for device
2. 0: 0, Unknown error: 0
?0%0[steffen@sherwood tmp]$ grep -F 25 /usr/include/sys/errno.h
#define ENOTTY 25 /* Inappropriate ioctl for device */
-- >8 --
So in fact the patches do what is necessary to make the changed
version act just as the plain systemcall.
> > - I favour haypos fullsync() approach
> Are you willing to update your patch accordingly?
Both patches still apply onto the tip of friday noon:
http://bugs.python.org/file22016/11877.9.diff,
http://bugs.python.org/file22046/11877-standalone.1.diff.
Again: i personally would favour os.fsync(fd, fullsync=True), because
that is the only way to put reliability onto unaware facilities
unaware (e.g. my S-Postman replaces os.fsync() with a special function
so that reliability is injected in- and onto Python's mailbox.py,
which calls plain os.fsync()), but i've been convinced that this is
impossible to do. It seems to be impossible to change os.fsync()
at all, because it has a standartized function prototype.
So what do you mean? Shall i rewrite 11877-standalone.1.diff to
always offer fullsync() whenever there is fsync()? This sounds to
be a useful change, because testing hasattr() of the one would
imply availability of the other.
> > + Aaarrg! I'm a liar!! I lie about - data integrity!!!
> Well, actually, some hard disks lie about this too :-)
Yeah. But hey:
"I feel save in New York City. I feel save in New York City."
Nice weekend - and may the juice be with you!
--Steffen
Ciao, sdaoden(*)(gmail.com)
ASCII ribbon campaign ( ) More nuclear fission plants
against HTML e-mailXcan serve more coloured
and proprietary attachments / \ and sounding animations
--
___
Python tracker
<http://bugs.python.org/issue11877>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11877] Change os.fsync() to support physical backing store syncs
Steffen Daode Nurpmeso added the comment:
> Are you willing to update your patch accordingly?
I'm a vain rooster! I've used fullfsync() in 11877-standalone.1.diff!
Sorry for that, haypo. :-/
11877.fullsync-1.diff uses fullsync() and will instead always be
provided when fsync() is available, to which it will fall back if
no special operating system functionality is available.
I really think this is the cleanest solution, because like this
a user can state "i want the strongest guarantees available on
data integrity", and Python does just that.
--Steffen
Ciao, sdaoden(*)(gmail.com)
ASCII ribbon campaign ( ) More nuclear fission plants
against HTML e-mailXcan serve more coloured
and proprietary attachments / \ and sounding animations
--
Added file: http://bugs.python.org/file22759/11877.fullsync-1.diff
___
Python tracker
<http://bugs.python.org/issue11877>
___diff --git a/Doc/library/os.rst b/Doc/library/os.rst
--- a/Doc/library/os.rst
+++ b/Doc/library/os.rst
@@ -706,6 +706,31 @@
Availability: Unix, and Windows starting in 2.2.3.
+.. function:: fullsync(fd)
+
+ The POSIX standart requires that :c:func:`fsync` must transfer the buffered
+ data to the storage device, not that the data is actually written by the
+ device itself. It explicitely leaves it up to operating system implementors
+ whether users are given stronger guarantees on data integrity or not. Some
+ systems also offer special functions which overtake the part of making such
+ stronger guarantees, i.e., Mac OS X and NetBSD. :func:`fullsync` is
+ identical to :func:`fsync` unless there is such special functionality
+ available, in which case that will be used.
+ To strive for best-possible data integrity, the following can be done::
+
+ # Force writeout of local buffer modifications
+ f.flush()
+ # Then synchronize the changes to physical backing store
+ if hasattr(os, 'fsync'):
+ os.fullsync(f.fileno())
+
+ ..note::
+ Calling this function may take a long time, since it may block
+ until the disk reports that the transfer has been completed.
+
+ Availability: See :func:`fsync`.
+
+
.. function:: ftruncate(fd, length)
Truncate the file corresponding to file descriptor *fd*, so that it is at
most
diff --git a/Lib/test/test_os.py b/Lib/test/test_os.py
--- a/Lib/test/test_os.py
+++ b/Lib/test/test_os.py
@@ -554,7 +554,7 @@
class TestInvalidFD(unittest.TestCase):
singles = ["fchdir", "fdopen", "dup", "fdatasync", "fstat",
- "fstatvfs", "fsync", "tcgetpgrp", "ttyname"]
+ "fstatvfs", "fsync", "fullsync", "tcgetpgrp", "ttyname"]
#singles.append("close")
#We omit close because it doesn'r raise an exception on some platforms
def get_single(f):
diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -1855,6 +1855,42 @@
{
return posix_fildes(fdobj, fsync);
}
+
+PyDoc_STRVAR(fullsync__doc__,
+"fullsync(fd)\n\n"
+"force write of file buffers to disk, and the flush of disk caches\n"
+"of the file given by file descriptor fd.");
+
+static PyObject *
+fullsync(PyObject *self, PyObject *fdobj)
+{
+/* See issue 11877 discussion */
+int res, fd = PyObject_AsFileDescriptor(fdobj);
+if (fd < 0)
+return NULL;
+if (!_PyVerify_fd(fd))
+return posix_error();
+
+Py_BEGIN_ALLOW_THREADS
+# if defined __APPLE__
+/* F_FULLFSYNC is not supported for all types of FDs/FSYSs;
+ * be on the safe side and test for inappropriate ioctl errors.
+ * Because plain fsync() may succeed even then, let it decide about error
*/
+res = fcntl(fd, F_FULLFSYNC);
+if (res < 0 && errno == ENOTTY)
+res = fsync(fd);
+# elif defined __NetBSD__
+res = fsync_range(fd, FFILESYNC | FDISKSYNC, 0, 0);
+# else
+res = fsync(fd);
+# endif
+Py_END_ALLOW_THREADS
+
+if (res < 0)
+return posix_error();
+Py_INCREF(Py_None);
+return Py_None;
+}
#endif /* HAVE_FSYNC */
#ifdef HAVE_FDATASYNC
@@ -8953,6 +8989,7 @@
#endif
#ifdef HAVE_FSYNC
{"fsync", posix_fsync, METH_O, posix_fsync__doc__},
+{"fullsync",fullsync, METH_O, fullsync__doc__},
#endif
#ifdef HAVE_FDATASYNC
{"fdatasync", posix_fdatasync, METH_O, posix_fdatasync__doc__},
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
Steffen Daode Nurpmeso added the comment: A sign! A sign! Someone with a name-name-name!! (Not a useful comment, i'm afraid.) -- nosy: +sdaoden ___ Python tracker <http://bugs.python.org/issue12730> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12730] Python's casemapping functions are untrustworthy due to narrow/wide build issues
Changes by Steffen Daode Nurpmeso : -- nosy: -sdaoden ___ Python tracker <http://bugs.python.org/issue12730> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Steffen Daode Nurpmeso added the comment: Heya. OpenBSD does support 1:1 threads via the RThread library since 2005, thanks to tedu@ and more fantastic guys! It about to be super-stable (one commit in 2011, last "real fix" in april 2010). (There is a techtalk from tedu@ (Ted Unangst) about this library on YouTube, btw.) -- nosy: +sdaoden ___ Python tracker <http://bugs.python.org/issue12868> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue12868] test_faulthandler.test_stack_overflow() failed on OpenBSD
Steffen Daode Nurpmeso added the comment: > In Python 3.3, you can use sys.thread_info to check which threading > library is used. Great! I didn't know that! -- ___ Python tracker <http://bugs.python.org/issue12868> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: I guess not at all. Well. -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: test_zlib.py (with my patch but that's somewhat identical in the end, say) does .s... -- Ran 37 tests in 1.809s OK (skipped=1) This is on Snow Leopard 64 bit, 02b70cb59701 (r88451) -> Python 3.3a0. Is there a switch i must trigger? Just pulled 24 changesets, recompiling and trying again with r88460. -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11285] io.py standart stream setup crash
New submission from Steffen Daode Nurpmeso : Just pulled 64380ee4bbc5 (r88500) and compiled, Python's busted: Fatal Python error: Py_Initialize: can't initialize sys standard streams Traceback (most recent call last): File "/Users/steffen/usr/opt/py3k/lib/python3.3/io.py", line 60, in Abort trap -- components: IO messages: 129065 nosy: sdaoden priority: normal severity: normal status: open title: io.py standart stream setup crash type: crash versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue11285> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: No, i've got no idea of this framework... Just did 'python3 test_zlib.py' directly. Thanks for the switch. But i can't test your thing due to issue11285, so this may take a while (others have more knowledge anyway).. (P.S.: your constant-folding stack patch is a great thing, just wanted to say this once..) -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: So here is this (with my patch, but this is for real: issue11277.2.patch): == CPython 3.3a0 (py3k, Feb 22 2011, 14:00:52) [GCC 4.2.1 (Apple Inc. build 5664)] == Darwin-10.6.0-i386-64bit little-endian == /private/var/folders/Da/DaZX3-k5G8a57zw6MSmjJTM/-Tmp-/test_python_89365 Testing with flags: sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0) [1/1] test_zlib test_adler32empty (test.test_zlib.ChecksumTestCase) ... ok test_adler32start (test.test_zlib.ChecksumTestCase) ... ok test_crc32_adler32_unsigned (test.test_zlib.ChecksumTestCase) ... ok test_crc32empty (test.test_zlib.ChecksumTestCase) ... ok test_crc32start (test.test_zlib.ChecksumTestCase) ... ok test_penguins (test.test_zlib.ChecksumTestCase) ... ok test_same_as_binascii_crc32 (test.test_zlib.ChecksumTestCase) ... ok test_badargs (test.test_zlib.ExceptionTestCase) ... ok test_badcompressobj (test.test_zlib.ExceptionTestCase) ... ok test_baddecompressobj (test.test_zlib.ExceptionTestCase) ... ok test_badlevel (test.test_zlib.ExceptionTestCase) ... ok test_decompressobj_badflush (test.test_zlib.ExceptionTestCase) ... ok test_big_compress_buffer (test.test_zlib.CompressTestCase) ... ok test_big_decompress_buffer (test.test_zlib.CompressTestCase) ... ok test_incomplete_stream (test.test_zlib.CompressTestCase) ... ok test_length_overflow (test.test_zlib.CompressTestCase) ... skipped 'not enough free memory, need at least 4 GB' test_speech (test.test_zlib.CompressTestCase) ... ok test_speech128 (test.test_zlib.CompressTestCase) ... ok test_badcompresscopy (test.test_zlib.CompressObjectTestCase) ... ok test_baddecompresscopy (test.test_zlib.CompressObjectTestCase) ... ok test_big_compress_buffer (test.test_zlib.CompressObjectTestCase) ... ok test_big_decompress_buffer (test.test_zlib.CompressObjectTestCase) ... ok test_compresscopy (test.test_zlib.CompressObjectTestCase) ... ok test_compressincremental (test.test_zlib.CompressObjectTestCase) ... ok test_compressoptions (test.test_zlib.CompressObjectTestCase) ... ok test_decompimax (test.test_zlib.CompressObjectTestCase) ... ok test_decompinc (test.test_zlib.CompressObjectTestCase) ... ok test_decompincflush (test.test_zlib.CompressObjectTestCase) ... ok test_decompress_incomplete_stream (test.test_zlib.CompressObjectTestCase) ... ok test_decompresscopy (test.test_zlib.CompressObjectTestCase) ... ok test_decompressmaxlen (test.test_zlib.CompressObjectTestCase) ... ok test_decompressmaxlenflush (test.test_zlib.CompressObjectTestCase) ... ok test_empty_flush (test.test_zlib.CompressObjectTestCase) ... ok test_flushes (test.test_zlib.CompressObjectTestCase) ... ok test_maxlenmisc (test.test_zlib.CompressObjectTestCase) ... ok test_odd_flush (test.test_zlib.CompressObjectTestCase) ... ok test_pair (test.test_zlib.CompressObjectTestCase) ... ok -- Ran 37 tests in 1.789s OK (skipped=1) 1 test OK. -- Added file: http://bugs.python.org/file20838/issue11277.2.patch ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11285] io.py standart stream setup crash
Steffen Daode Nurpmeso added the comment: Let me see some minutes. -- ___ Python tracker <http://bugs.python.org/issue11285> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file20837/issue11277.patch ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: (Is not that much help for a >4GB error, huh?) -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: Just stepping ... with c8d1f99f25eb/r88476: == CPython 3.3a0 (py3k, Feb 22 2011, 14:18:19) [GCC 4.2.1 (Apple Inc. build 5664)] == Darwin-10.6.0-i386-64bit little-endian == /private/var/folders/Da/DaZX3-k5G8a57zw6MSmjJTM/-Tmp-/test_python_5126 Testing with flags: sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0) [1/1] test_zlib test_adler32empty (test.test_zlib.ChecksumTestCase) ... ok test_adler32start (test.test_zlib.ChecksumTestCase) ... ok test_crc32_adler32_unsigned (test.test_zlib.ChecksumTestCase) ... ok test_crc32empty (test.test_zlib.ChecksumTestCase) ... ok test_crc32start (test.test_zlib.ChecksumTestCase) ... ok test_penguins (test.test_zlib.ChecksumTestCase) ... ok test_same_as_binascii_crc32 (test.test_zlib.ChecksumTestCase) ... ok test_big_buffer (test.test_zlib.ChecksumBigBufferTestCase) ... ^C ^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C Bus error -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11285] io.py standart stream setup crash
Steffen Daode Nurpmeso added the comment: According to adorable but slow Mercurial: The first bad revision is: changeset: 9894:fc8a94cc17a4 branch: py3k user:raymond.hettinger date:Tue Feb 22 01:41:50 2011 +0100 summary: [svn r88490] Issue #11085: Moved collections abstract base classes into a separate module -- ___ Python tracker <http://bugs.python.org/issue11285> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment:
.. even with a self-compiled 1.2.3, INT_MAX/1000 ... nothing.
The problem is not crc32(), but the buffer itself:
if (pbuf.len > 1024*5) {
unsigned char *buf = pbuf.buf;
Py_ssize_t len = pbuf.len;
Py_ssize_t i;
fprintf(stderr, "CRC 32 2.1\n");
for(i=0; (size_t)i < (size_t)len;++i)
*buf++ = 1;
fprintf(stderr, "CRC 32 2.2\n");
2.2 is never reached (in fact accessing buf[1] already causes fault).
Thus the problem is not zlib, but PyArg_ParseTuple().
But just don't ask me more on that!
--
___
Python tracker
<http://bugs.python.org/issue11277>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: (P.S.: of course talking about ChecksumBigBufferTestCase and the 4GB, say.) -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment:
Snippet
if (pbuf.len > 1024*5) {
volatile unsigned char *buf = pbuf.buf;
Py_ssize_t len = pbuf.len;
Py_ssize_t i = 0;
volatile unsigned char au[100];
volatile unsigned char*x = au;
fprintf(stderr, "CRC ENTER, buffer=%p\n", buf);
for (i=0; (size_t)i < (size_t)len; ++i) {
fprintf(stderr, "%ld, buf=%p\n", (signed long)i, buf);
*x = *buf++;
}
results in
test_big_buffer (test.test_zlib.ChecksumBigBufferTestCase) ... CRC ENTER,
buffer=0x1014ab000
0, buf=0x1014ab000
--
___
Python tracker
<http://bugs.python.org/issue11277>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: I have a MacBook with 2 GB RAM. Of course i'm a little bit messy, so an entry is half written before it comes to an ... end. msg129091 is real life, though. Antoine, your msg129093 patch of test_zlib.py does it (with and without fprintf(3)s). CRC ok etc., it just works. (Seems mmap(2) has a problem here, i would say; the mentioned bug report is from 2003, so the golden sunset watchers may need some more additional time, if you allow me that comment.) -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: (That is to say: i think it's better not to assume that these boys plan to *ever* fix it. (Though mmap(2) is not CoreAudio/AudioUnit.)) -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: (neologix: SIGBUS is not the same as SIGSEGV. You know. Thanks for this nice bug report. Eight years is a .. time in computer programming - unbelievable, thinking of all these nervous wrecks who ever reported a bug to Apple! Man!!!) -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: neologix: even with 2 GB RAM top(1) shows more than 600 MB free memory with the 4 GB test up and running ... in an Mac OS X environment ... Lucky me, i don't believe them a single word... -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11285] io.py standart stream setup crash
Steffen Daode Nurpmeso added the comment: Hmm. Note that this problem does *not* occur if i don't install Python but run it in place, e.g. 'cd Lib/test; ../../python.exe -m test -v -uall test_iter' works just perfect. -- ___ Python tracker <http://bugs.python.org/issue11285> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11285] io.py standart stream setup crash
Steffen Daode Nurpmeso added the comment: ... and the problem is all gone with 8c2935f180fa/r88525. So i'm faitful now and close this early alpha-stage problem. One of the nosy ones may add a nice description, re-open it or so. -- ___ Python tracker <http://bugs.python.org/issue11285> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11285] io.py standart stream setup crash
Steffen Daode Nurpmeso added the comment: (.. should close it, then.) -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue11285> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: I append a doc_lib_mmap.patch which may be helpful for those poor creatures who plan to write Python scripts for Mac OS X. (It may be a useful add-on anyway.) -- Added file: http://bugs.python.org/file20859/doc_lib_mmap.patch ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file20859/doc_lib_mmap.patch ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: Sorry, i've got that kid running around which sometimes doesn't know what it is doing. But this documentation patch may really be a help. It's my first doc-patch, so it surely needs to be revised, if interest exists in such a patch for mmap at all, say. Thanks for your understanding. -- Added file: http://bugs.python.org/file20861/doc_lib_mmap.patch ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11300] mmap() large file failures on Mac OS X docfix
New submission from Steffen Daode Nurpmeso : Issue 11277 was closed upon the generally accepted conclusion that the failure is that mmap(2) on Mac OS X has a long time known bug (see msg129107, or directly http://lists.apple.com/archives/darwin-development/2003/Jun/msg00141.html). Issue 11277 fixed the problem in the test suite, but it does give end-users no hint that such a problem exists (on the documentation side). And it's my believe that a beginner or even a non-low-level programmer may be lost in an equal situation. (Though one could argue that such a HE should not use mmap(2), but everyone has a first-time somewhen, say.) In addition, there are many people who don't have the chance to work/test on multiple operating systems and/or hardware, and who need the hand of people who are experienced and can give hints on stupidiness or however this word is correctly spelled. (Am i preaching already?) Because Python is expected to be a user-friendly language i do request a documentation patch which gives a hint. I'll apply a patch, but i'm somewhat new to Python and rST etc., so it surely needs a hand. (Maybe even its speech is too aggressive??) -- assignee: docs@python components: Documentation files: doc_lib_mmap.patch keywords: patch messages: 129221 nosy: docs@python, sdaoden priority: normal severity: normal status: open title: mmap() large file failures on Mac OS X docfix type: feature request versions: Python 3.2, Python 3.3 Added file: http://bugs.python.org/file20867/doc_lib_mmap.patch ___ Python tracker <http://bugs.python.org/issue11300> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment:
I wonder what this normalize_encoding() does! Here is a pretty standard
version of mine which is a bit more expensive but catches match more cases!
This is stripped, of course, and can be rewritten very easily to Python's needs
(i.e. using char[32] instead of char[11].
* @@li If a character is either ::s_char_is_space() or ::s_char_is_punct():
* @@liReplace with ASCII space (0x20).
* @@liSqueeze adjacent spaces to a single one.
* @@li Else if a character is ::s_char_is_alnum():
* @@li::s_char_to_lower() characters.
* @@liSeparate groups of alphas and digits with ASCII space (0x20).
* @@li Else discard character.
* E.g. "ISO_8859---1" becomes "iso 8859 1"
* and "ISO8859-1" also becomes "iso 8859 1".
s_textcodec_normalize_name(s_CString *_name) {
enum { C_NONE, C_WS, C_ALPHA, C_DIGIT } c_type = C_NONE;
char *name, c;
auto s_CString input;
s_cstring_swap(s_cstring_init(&input), _name);
_name = s_cstring_reserve(_name, 31, s_FAL0);
name = s_cstring_cstr(&input);
while ((c = *(name++)) != s_NUL) {
s_si8 sep = s_FAL0;
if (s_char_is_space(c) || s_char_is_punct(c)) {
if (c_type == C_WS)
continue;
c_type = C_WS;
c = ' ';
} else if (s_char_is_alpha(c)) {
sep = (c_type == C_DIGIT);
c_type = C_ALPHA;
c = s_char_to_lower(c);
} else if (s_char_is_digit(c)) {
sep = (c_type == C_ALPHA);
c_type = C_DIGIT;
} else
continue;
do
_name = s_cstring_append_char(_name, (sep ? ' ' : c));
while (--sep >= s_FAL0);
}
s_cstring_destroy(&input);
return _name;
}
--
nosy: +sdaoden
___
Python tracker
<http://bugs.python.org/issue11303>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: (That is to say, i would do it. But not if _cpython is thrown to trash ,-); i.e. not if there is not a slight chance that it gets actually patched in because this performance issue probably doesn't mean a thing in real life. You know, i'm a slow programmer, i would need *at least* two hours to rewrite that in plain C in a way that can make it as a replacement of normalize_encoding().) -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: .. i don't have actually invented this algorithm (but don't ask me where i got the idea from years ago), i've just implemented the function you see. The algorithm itself avoids some pitfalls in respect to combining numerics and significantly reduces the number of possible normalization cases: "ISO-8859-1", "ISO8859-1", "ISO_8859-1", "LATIN1" (+ think of additional mispellings) all become "iso 8859 1", "latin 1" in the end -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: (Everything else is beyond my scope. But normalizing _ to - is possibly a bad idea as far as i can remember the situation three years ago.) -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: P.P.S.: separating alphanumerics is a win for things like, e.g. UTF-16BE: it gets 'utf 16 be' - think about the possible mispellings here and you see this algorithm is a good thing -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: So, well, a-ha, i will boot my laptop this evening and (try to) write a patch for normalize_encoding(), which will match the standart conforming LATIN1 and also will continue to support the illegal latin-1 without actually changing the two users PyUnicode_Decode() and PyUnicode_AsEncodedString(), from which i better keep the hands off. But i'm slow, it may take until tomorrow... -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: So happy hacker haypo did it, different however. It's illegal, but since this is a static function which only serves some specific internal strcmp(3)s it may do for the mentioned charsets. I won't boot my laptop this evening. -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: That's ok by me. And 'happy hacker haypo' was not ment unfriendly, i've only repeated the first response i've ever posted back to this tracker (guess who was very fast at that time :)). -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: (Not issue related) Ezio and Alexander: after reading your posts and looking back on my code: you're absolutely right. Doing resize(31) is pointless: it doesn't save space (mempool serves [8],16,24,32 there; and: dynamic, normalized coded names don't exist that long in real life, too). And append_char() is inlined but much more expensive than doing (register-loaded) *(target++)=char. Thus i now do believe my code is a bug and i will rewrite doing *target=cstr(resize(len(input)*2)) ... truncate() instead! Thanks. -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11322] encoding package's normalize_encoding() function is too slow
Changes by Steffen Daode Nurpmeso : -- nosy: +sdaoden ___ Python tracker <http://bugs.python.org/issue11322> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment: 11:12 ~/tmp $ python3 ~/usr/opt/py3k/lib/python3.3/test_zlib.py Bus error Your code snippet: 11:21 ~/tmp $ /usr/bin/time -lp python3 test.py posix.stat_result(st_mode=33184, st_ino=10066605, st_dev=234881025, st_nlink=1, st_uid=502, st_gid=20, st_size=4294967300, st_atime=1298715813, st_mtime=1298715813, st_ctime=1298715813) posix.stat_result(st_mode=33184, st_ino=10066605, st_dev=234881025, st_nlink=1, st_uid=502, st_gid=20, st_size=4294967300, st_atime=1298715813, st_mtime=1298715813, st_ctime=1298715813) real71.66 user 0.06 sys 3.71 0 maximum resident set size 0 average shared memory size 0 average unshared data size 0 average unshared stack size 0 page reclaims 0 page faults 0 swaps 0 block input operations 57 block output operations 0 messages sent 0 messages received 0 signals received 2112 voluntary context switches 0 involuntary context switches On Fri, Feb 25, 2011 at 05:05:19PM +, Charles-Francois Natali wrote: > >Charles-Francois Natali added the comment: > >Could you try with this: > > def setUp(self): > with open(support.TESTFN, "wb+") as f: > f.seek(_4G) > f.write(b"asdf") > f.flush() >+ os.fsync(f.fileno()) > self.mapping = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ) > >HFS+ doesn't seem to support sparse files, so the file is actually >zero-filled asynchronously. >Maybe the mapping gets done before the blocks have been allocated, >which triggers a segfault when the first page is accessed. >I'm not sure it'll make any difference, but I'm curious... > >Also, I'd be curious to see the result of > >""" >import os > >name = '/tmp/foo' >f = open(name, 'wb') >f.seek(1 << 32) >f.write(b'asdf') >f.flush() >print(os.fstat(f.fileno())) >f.close() >print(os.stat(name)) >""" > >Thanks ! > >-- > >___ >Python tracker ><http://bugs.python.org/issue11277> >___ -- ___ Python tracker <http://bugs.python.org/issue11277> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11277] test_zlib crashes under Snow Leopard buildbot
Steffen Daode Nurpmeso added the comment:
I'll give you the same result again but with additional clock(),
just for a heart's pleasure:
clock(): 0.100958 , fstat(): posix.stat_result(st_mode=33184, st_ino=10075508,
st_dev=234881025, st_nlink=1, st_uid=502, st_gid=20, st_size=4294967300,
st_atime=1298719201, st_mtime=1298719305, st_ctime=1298719305)
> f.close()
> print('clock():', time.clock(), ', stat():', os.stat(name))
clock(): 3.75792 , stat(): posix.stat_result(st_mode=33184, st_ino=10075508,
st_dev=234881025, st_nlink=1, st_uid=502, st_gid=20, st_size=4294967300,
st_atime=1298719201, st_mtime=1298719305, st_ctime=1298719305)
Please don't assume i go for Mac OS X ...
In the end you *always* need to implement an expensive state
machine to get around long-known bugs, mis-implementations or
other poops there.
--
___
Python tracker
<http://bugs.python.org/issue11277>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11303] b'x'.decode('latin1') is much slower than b'x'.decode('latin-1')
Steffen Daode Nurpmeso added the comment: On Fri, Feb 25, 2011 at 03:43:06PM +, Marc-Andre Lemburg wrote: > > Marc-Andre Lemburg added the comment: > > r88586: Normalized the encoding names for Latin-1 and UTF-8 to > 'latin-1' and 'utf-8' in the stdlib. Even though - or maybe exactly because - i'm a newbie, i really want to add another message after all this biting is over. I've just read PEP 100 and msg129257 (on Issue 5902), and i feel a bit confused. > Marc-Andre Lemburg added the comment: > It turns out that there are three "normalize" functions that are > successively applied to the encoding name during evaluation of > str.encode/str.decode. > > 1. normalize_encoding() in unicodeobject.c > > This was added to have the few shortcuts we have in the C code > for commonly used codecs match more encoding aliases. > > The shortcuts completely bypass the codec registry and also > bypass the function call overhead incurred by codecs > run via the codec registry. The thing that i don't understand the most is that illegal (according to IANA standarts) names are good on the one hand (latin-1, utf-16-be), but bad on the other, i.e. in my group-preserving code or haypos very fast but name-joining patch (the first): a *local* change in unicodeobject.c, which' result is *only* used for the two users PyUnicode_Decode() and PyUnicode_AsEncodedString(). However: > Marc-Andre Lemburg added the comment: > Programmers who don't use the encoding names triggering those > optimizations will still have a running program, it'll only be > a bit slower and that's perfectly fine. > Marc-Andre Lemburg added the comment: > think rather than removing any hyphens, spaces, etc. the > function should additionally: > > * add hyphens whenever (they are missing and) there's switch > from [a-z] to [0-9] > > That way you end up with the correct names for the given set > of optimized encoding names. haypos patch can easily be adjusted to reflect this, resulting in a much cleaner code in the two mentioned users, because normalize_encoding() did the job it was ment for. (Hmmm, and my own code could also be adjusted to match Python semantics (using hyphen instead of space as a group-separator), so that an end-user has the choice in between *all* IANA standart names (e.g. "ISO-8859-1", "ISO8859-1", "ISO_8859-1", "LATIN1"), and would gain the full optimization benefit of using latin-1, which seems to be pretty useful for limburger.) > Ezio Melotti wrote: > Marc-Andre Lemburg wrote: >> That won't work, Victor, since it makes invalid encoding >> names valid, e.g. 'utf(=)-8'. > > That already works in Python (thanks to encodings.normalize_encoding) *However*: in PEP 100 Python has decided to go its own way a decade ago. > Marc-Andre Lemburg added the comment: > 2. normalizestring() in codecs.c > > This is the normalization applied by the codec registry. See PEP 100 > for details: > > """ >Search functions are expected to take one argument, >the encoding name in all lower case letters and with hyphens >and spaces converted to underscores, ... > """ > 3. normalize_encoding() in encodings/__init__.py > > This is part of the stdlib encodings package's codec search function. First: *i* go for haypo: > It's funny: we have 3 functions to normalize an encoding name, and > each function does something else :-) (that's Issue 11322:) > We should first implement the same algorithm of the 3 normalization > functions and add tests for them And *i* don't understand anything else (*i* do have *my* - now furtherly optimized, thanks - s_textcodec_normalize_name()). However, two different ones (very fast thing which is enough to meet unicodeobject.c and a global one for anything else) may also do. Isn't anything else a maintenance mess? Where is that database, are there any known dependencies which are exposed to end-users? Or the like. I'm much too loud, and have a nice weekend. -- ___ Python tracker <http://bugs.python.org/issue11303> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] (test_os) os.sendfile() error
New submission from Steffen Daode Nurpmeso : Antoine, i made you noisy because you took care about Issue11323. I'm at r88671 (271057c7c6f3). I'm opening this because the buildbot URL (thanks) doesn't show anything about sendfile(2). 12:47 ~/arena/code.extern.repos/py3k.hg $ ./python.exe -Wd -m test.test_os -r -w -v ... test_headers (__main__.TestSendfile) ... FAIL test_invalid_offset (__main__.TestSendfile) ... ok test_offset_overflow (__main__.TestSendfile) ... ok test_send_at_certain_offset (__main__.TestSendfile) ... ok test_send_whole_file (__main__.TestSendfile) ... ok test_trailers (__main__.TestSendfile) ... /Users/steffen/arena/code.extern.repos/py3k.hg/Lib/unittest/case.py:387: ResourceWarning: unclosed file <_io.BufferedReader name='@test_28253_tmp2'> function() ok test_set_get_priority (__main__.ProgramPriorityTests) ... ok == FAIL: test_headers (__main__.TestSendfile) -- Traceback (most recent call last): File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/test/test_os.py", line 1517, in test_headers self.assertEqual(total_sent, len(expected_data)) AssertionError: 163840 != 164352 -- Ran 92 tests in 0.447s FAILED (failures=1, skipped=10) Traceback (most recent call last): File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/runpy.py", line 160, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/runpy.py", line 73, in _run_code exec(code, run_globals) File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/test/test_os.py", line 1573, in test_main() File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/test/test_os.py", line 1569, in test_main ProgramPriorityTests, File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/test/support.py", line 1187, in run_unittest _run_suite(suite) File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/test/support.py", line 1170, in _run_suite raise TestFailed(err) test.support.TestFailed: Traceback (most recent call last): File "/Users/steffen/arena/code.extern.repos/py3k.hg/Lib/test/test_os.py", line 1517, in test_headers self.assertEqual(total_send, len(expected_data)) AssertionError: 163840 != 164352 -- components: Library (Lib) messages: 129689 nosy: pitrou, sdaoden priority: normal severity: normal status: open title: (test_os) os.sendfile() error type: behavior versions: Python 3.3 ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] (test_os) os.sendfile() error
Steffen Daode Nurpmeso added the comment:
It's about the headers[]:
def test_headers(self):
total_sent = 0
sent = os.sendfile(self.sockno, self.fileno, 0, 4096,
headers=[b"x" * 512])
total_sent += sent
print("HEADER SENT:", sent)
prints
test_headers (__main__.TestSendfile) ... HEADER SENT: 4096
Well. 'man 2 sendfile' tells me it is supporting headers and
trailers since Darwin 9.0 aka Mac OS X 10.5.
--
___
Python tracker
<http://bugs.python.org/issue11351>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] (test_os) os.sendfile() error
Steffen Daode Nurpmeso added the comment: posixmodule.c: iov_setup(): IOV[i]: buffer:0x10152b8f0, len:512 iov_setup() leave ok before sendfile: sf=0x7fff5fbfb410, sf.headers=0x10036c780 sf.headers[0].iov_base=0x10152d950, .iov_len=512 after sendfile: sbytes=4096 Maybe Mac OS X does not support headers (i'm on Mac OS X 10.6)? -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] (test_os) os.sendfile() error
Steffen Daode Nurpmeso added the comment:
Did you know that 'sbytes' is not adjusted to match possibly
existent headers and trailers in posixmodule.c?
This however is required according to Mac OS X 'man 2 sendfile'.
I'll attach a simple, naive patch for a very intelligent and
outstanding operating system.
--
Added file: http://bugs.python.org/file20943/posixmodule.naive-apple-patch
___
Python tracker
<http://bugs.python.org/issue11351>
___diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -5867,9 +5867,10 @@
#ifdef HAVE_SENDFILE
#if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
-static int
+static off_t
iov_setup(struct iovec **iov, Py_buffer **buf, PyObject *seq, int cnt, int
type)
{
+off_t ret = 0;
int i, j;
*iov = PyMem_New(struct iovec, cnt);
if (*iov == NULL) {
@@ -5880,7 +5881,7 @@
if (*buf == NULL) {
PyMem_Del(*iov);
PyErr_NoMemory();
-return 0;
+return ret;
}
for (i = 0; i < cnt; i++) {
@@ -5889,14 +5890,18 @@
PyMem_Del(*iov);
for (j = 0; j < i; j++) {
PyBuffer_Release(&(*buf)[j]);
- }
+}
PyMem_Del(*buf);
-return 0;
+ret ^= ret;
+return ret;
}
(*iov)[i].iov_base = (*buf)[i].buf;
-(*iov)[i].iov_len = (*buf)[i].len;
-}
-return 1;
+do {off_t x = (*buf)[i].len;
+(*iov)[i].iov_len = x;
+ret += x;
+} while (0);
+}
+return ret;
}
static void
@@ -5954,10 +5959,15 @@
"sendfile() headers must be a sequence or None");
return NULL;
} else {
+off_t i = 0; /* Uninitialized warning */
sf.hdr_cnt = PySequence_Size(headers);
-if (sf.hdr_cnt > 0 && !iov_setup(&(sf.headers), &hbuf,
-headers, sf.hdr_cnt, PyBUF_SIMPLE))
+if (sf.hdr_cnt > 0 &&
+!(i = iov_setup(&(sf.headers), &hbuf,
+headers, sf.hdr_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
if (trailers != NULL) {
@@ -5966,10 +5976,15 @@
"sendfile() trailers must be a sequence or None");
return NULL;
} else {
+off_t i = 0; /* Uninitialized warning */
sf.trl_cnt = PySequence_Size(trailers);
-if (sf.trl_cnt > 0 && !iov_setup(&(sf.trailers), &tbuf,
-trailers, sf.trl_cnt, PyBUF_SIMPLE))
+if (sf.trl_cnt > 0 &&
+!(i = iov_setup(&(sf.trailers), &tbuf,
+trailers, sf.trl_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Changes by Steffen Daode Nurpmeso : -- title: (test_os) os.sendfile() error -> Mac OS X os.sendfile() ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] (test_os) os.sendfile() error
Steffen Daode Nurpmeso added the comment: Before i'm going down, here is the paragraph from 'man 2 sendfile': When a header or trailer is specified the value of len returned will include the size of header or trailer sent. The user should provide suf- ficiently large value of len as argument including the size of header or trailer, otherwise only part of file data will be sent following the header. -- title: Mac OS X os.sendfile() -> (test_os) os.sendfile() error ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: (But see msg12970. And the test runs fine with my patch applied.) On Mon, Feb 28, 2011 at 02:17:46PM +, Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > > Likely explanation is http://bugs.python.org/issue10882#msg129555. > > -- > nosy: +giampaolo.rodola > > ___ > Python tracker > <http://bugs.python.org/issue11351> > ___ -- title: (test_os) os.sendfile() error -> Mac OS X os.sendfile() ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: (Should have been msg129706, sorry.) P.S.: yes, you're right. But now: issue 10882 is on adding os.sendfile(), not on fixing errors on os.sendfile(). ;-) -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment:
Will this do? Otherwise feel free to adjust the patch the way
Python needs it, i'll learn from that, then, for possible future
things. (I need to go down now.)
On Mon, Feb 28, 2011 at 02:37:16PM +, Antoine Pitrou wrote:
>
> Antoine Pitrou added the comment:
>
> Your patch is not cleaned up. There are strange things like:
>
> +ret ^= ret;
>
> (while would you xor an off_t?)
>
> and
>
> +do {off_t x = (*buf)[i].len;
> +(*iov)[i].iov_len = x;
> +ret += x;
> +} while (0);
--
keywords: +patch
Added file: http://bugs.python.org/file20946/issue11351.patch
___
Python tracker
<http://bugs.python.org/issue11351>
___diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -5867,10 +5867,11 @@
#ifdef HAVE_SENDFILE
#if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
-static int
+static off_t
iov_setup(struct iovec **iov, Py_buffer **buf, PyObject *seq, int cnt, int
type)
{
int i, j;
+off_t len, ret = 0;
*iov = PyMem_New(struct iovec, cnt);
if (*iov == NULL) {
PyErr_NoMemory();
@@ -5880,7 +5881,7 @@
if (*buf == NULL) {
PyMem_Del(*iov);
PyErr_NoMemory();
-return 0;
+return ret;
}
for (i = 0; i < cnt; i++) {
@@ -5889,14 +5890,17 @@
PyMem_Del(*iov);
for (j = 0; j < i; j++) {
PyBuffer_Release(&(*buf)[j]);
- }
+}
PyMem_Del(*buf);
-return 0;
+ret = 0;
+return ret;
}
(*iov)[i].iov_base = (*buf)[i].buf;
-(*iov)[i].iov_len = (*buf)[i].len;
-}
-return 1;
+len = (*buf)[i].len;
+(*iov)[i].iov_len = len;
+ret += len;
+}
+return ret;
}
static void
@@ -5954,10 +5958,15 @@
"sendfile() headers must be a sequence or None");
return NULL;
} else {
+off_t i = 0; /* Uninitialized warning */
sf.hdr_cnt = PySequence_Size(headers);
-if (sf.hdr_cnt > 0 && !iov_setup(&(sf.headers), &hbuf,
-headers, sf.hdr_cnt, PyBUF_SIMPLE))
+if (sf.hdr_cnt > 0 &&
+!(i = iov_setup(&(sf.headers), &hbuf,
+headers, sf.hdr_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
if (trailers != NULL) {
@@ -5966,10 +5975,15 @@
"sendfile() trailers must be a sequence or None");
return NULL;
} else {
+off_t i = 0; /* Uninitialized warning */
sf.trl_cnt = PySequence_Size(trailers);
-if (sf.trl_cnt > 0 && !iov_setup(&(sf.trailers), &tbuf,
-trailers, sf.trl_cnt, PyBUF_SIMPLE))
+if (sf.trl_cnt > 0 &&
+!(i = iov_setup(&(sf.trailers), &tbuf,
+trailers, sf.trl_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment:
This version uses Py_ssize_t for the lengths and seems to be more
Python-style beside that, too.
It compiles clean and the test is ok.
--
Added file: http://bugs.python.org/file20949/issue11351-2.patch
___
Python tracker
<http://bugs.python.org/issue11351>
___diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -5867,36 +5867,42 @@
#ifdef HAVE_SENDFILE
#if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
-static int
+static Py_ssize_t
iov_setup(struct iovec **iov, Py_buffer **buf, PyObject *seq, int cnt, int
type)
{
int i, j;
+Py_ssize_t blen, total = 0;
+
*iov = PyMem_New(struct iovec, cnt);
if (*iov == NULL) {
PyErr_NoMemory();
-return 0;
-}
+return total;
+}
+
*buf = PyMem_New(Py_buffer, cnt);
if (*buf == NULL) {
PyMem_Del(*iov);
PyErr_NoMemory();
-return 0;
+return total;
}
for (i = 0; i < cnt; i++) {
-if (PyObject_GetBuffer(PySequence_GetItem(seq, i), &(*buf)[i],
-type) == -1) {
+if (PyObject_GetBuffer(PySequence_GetItem(seq, i),
+ &(*buf)[i], type) == -1) {
PyMem_Del(*iov);
for (j = 0; j < i; j++) {
PyBuffer_Release(&(*buf)[j]);
- }
+}
PyMem_Del(*buf);
-return 0;
+total = 0;
+return total;
}
(*iov)[i].iov_base = (*buf)[i].buf;
-(*iov)[i].iov_len = (*buf)[i].len;
-}
-return 1;
+blen = (*buf)[i].len;
+(*iov)[i].iov_len = blen;
+total += len;
+}
+return total;
}
static void
@@ -5954,10 +5960,15 @@
"sendfile() headers must be a sequence or None");
return NULL;
} else {
+Py_ssize_t i = 0; /* Avoid uninitialized warning */
sf.hdr_cnt = PySequence_Size(headers);
-if (sf.hdr_cnt > 0 && !iov_setup(&(sf.headers), &hbuf,
-headers, sf.hdr_cnt, PyBUF_SIMPLE))
+if (sf.hdr_cnt > 0 &&
+!(i = iov_setup(&(sf.headers), &hbuf,
+headers, sf.hdr_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
if (trailers != NULL) {
@@ -5966,10 +5977,15 @@
"sendfile() trailers must be a sequence or None");
return NULL;
} else {
+Py_ssize_t i = 0; /* Avoid uninitialized warning */
sf.trl_cnt = PySequence_Size(trailers);
-if (sf.trl_cnt > 0 && !iov_setup(&(sf.trailers), &tbuf,
-trailers, sf.trl_cnt, PyBUF_SIMPLE))
+if (sf.trl_cnt > 0 &&
+!(i = iov_setup(&(sf.trailers), &tbuf,
+trailers, sf.trl_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment:
I'm trashing your mailbox, but
http://bugs.python.org/file20949/issue11351-2.patch should do
'total+=blen' not 'total+=len'.
Anyway i attach yet a third patch which does some code cleanup in
general and is a bit more of my very own coding style, because
i have seen that gotos seem to be possible in Python, and because
i was unable to enjoy the code (and this is beautiful C!) before.
It also follows PEP guidelines more than before.
This compiles clean and runs the test even if i
hg revert && hg patch
(I'm a bit messy)
--
Added file: http://bugs.python.org/file20955/issue11351-3.patch
___
Python tracker
<http://bugs.python.org/issue11351>
___diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -5866,37 +5866,47 @@
}
#ifdef HAVE_SENDFILE
-#if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
-static int
+# if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
+static Py_ssize_t
iov_setup(struct iovec **iov, Py_buffer **buf, PyObject *seq, int cnt, int
type)
{
int i, j;
+Py_ssize_t blen, total = 0;
+
*iov = PyMem_New(struct iovec, cnt);
if (*iov == NULL) {
PyErr_NoMemory();
-return 0;
-}
+goto jleave;
+}
+
*buf = PyMem_New(Py_buffer, cnt);
if (*buf == NULL) {
PyMem_Del(*iov);
PyErr_NoMemory();
-return 0;
+goto jleave;
}
for (i = 0; i < cnt; i++) {
-if (PyObject_GetBuffer(PySequence_GetItem(seq, i), &(*buf)[i],
-type) == -1) {
-PyMem_Del(*iov);
-for (j = 0; j < i; j++) {
-PyBuffer_Release(&(*buf)[j]);
- }
-PyMem_Del(*buf);
-return 0;
-}
+if (PyObject_GetBuffer(PySequence_GetItem(seq, i),
+ &(*buf)[i], type) == -1)
+goto jrelease;
+
(*iov)[i].iov_base = (*buf)[i].buf;
-(*iov)[i].iov_len = (*buf)[i].len;
-}
-return 1;
+blen = (*buf)[i].len;
+(*iov)[i].iov_len = blen;
+total += blen;
+}
+
+jleave:
+return total;
+
+jrelease:
+PyMem_Del(*iov);
+for (j = 0; j < i; j++)
+PyBuffer_Release(&(*buf)[j]);
+PyMem_Del(*buf);
+total = 0;
+goto jleave;
}
static void
@@ -5904,12 +5914,11 @@
{
int i;
PyMem_Del(iov);
-for (i = 0; i < cnt; i++) {
+for (i = 0; i < cnt; i++)
PyBuffer_Release(&buf[i]);
-}
PyMem_Del(buf);
}
-#endif
+# endif
PyDoc_STRVAR(posix_sendfile__doc__,
"sendfile(out, in, offset, nbytes) -> byteswritten\n\
@@ -5924,10 +5933,10 @@
Py_ssize_t ret;
off_t offset;
-#if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
-#ifndef __APPLE__
+# if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
+# ifndef __APPLE__
Py_ssize_t len;
-#endif
+# endif
PyObject *headers = NULL, *trailers = NULL;
Py_buffer *hbuf, *tbuf;
off_t sbytes;
@@ -5939,46 +5948,58 @@
"offset", "count",
"headers", "trailers", "flags", NULL};
-#ifdef __APPLE__
if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiO&O&|OOi:sendfile",
-keywords, &out, &in, _parse_off_t, &offset, _parse_off_t, &sbytes,
-#else
-if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiO&n|OOi:sendfile",
-keywords, &out, &in, _parse_off_t, &offset, &len,
-#endif
-&headers, &trailers, &flags))
+ keywords, &out, &in, _parse_off_t,
+# ifdef __APPLE__
+ &offset, _parse_off_t, &sbytes,
+# else
+ &offset, &len,
+# endif
+ &headers, &trailers, &flags))
return NULL;
+
if (headers != NULL) {
if (!PySequence_Check(headers)) {
PyErr_SetString(PyExc_TypeError,
-"sendfile() headers must be a sequence or None");
+"sendfile() headers must be a sequence or None");
return NULL;
} else {
+Py_ssize_t i = 0; /* (Avoid uninitialized warning) */
sf.hdr_cnt = PySequence_Size(headers);
-if (sf.hdr_cnt > 0 && !iov_setup(&(sf.headers), &hbuf,
-headers, sf.hdr_cnt, PyBUF_SIMPLE))
+if (sf.hdr_cnt > 0 &&
+(i = iov_setup(&(sf.headers), &hbuf,
+
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: Aah, unfortunately - what a shame, this beautiful language ... But do i need to re-post the real (ouch!) issue11351-2.patch, i.e. with 'total+=blen' instead of 'total+=len'? Or is it sufficient now with these comments (for Giampaolo')? > Antoine Pitrou added the comment: > > Well the third patch has a lot of spurious style changes that make it harder > to spot the actual changes. -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: Maybe Giampaolo' should at least replace these two PyArg_ParseTupleAndKeywords() near (unpatched) line 5942, because doing it like this is messes up (at least) vim(1) group detection, resulting in a lot of bold red mis-brace warnings further on? -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment:
So this is fixed -2.patch supplemented only with
PyArg_ParseTupleAndKeywords() fixed in a way that re-enables
correct syntax highlighting.
--
Added file: http://bugs.python.org/file20959/issue11351-4.patch
___
Python tracker
<http://bugs.python.org/issue11351>
___diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -5867,36 +5867,42 @@
#ifdef HAVE_SENDFILE
#if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
-static int
+static Py_ssize_t
iov_setup(struct iovec **iov, Py_buffer **buf, PyObject *seq, int cnt, int
type)
{
int i, j;
+Py_ssize_t blen, total = 0;
+
*iov = PyMem_New(struct iovec, cnt);
if (*iov == NULL) {
PyErr_NoMemory();
-return 0;
-}
+return total;
+}
+
*buf = PyMem_New(Py_buffer, cnt);
if (*buf == NULL) {
PyMem_Del(*iov);
PyErr_NoMemory();
-return 0;
+return total;
}
for (i = 0; i < cnt; i++) {
-if (PyObject_GetBuffer(PySequence_GetItem(seq, i), &(*buf)[i],
-type) == -1) {
+if (PyObject_GetBuffer(PySequence_GetItem(seq, i),
+ &(*buf)[i], type) == -1) {
PyMem_Del(*iov);
for (j = 0; j < i; j++) {
PyBuffer_Release(&(*buf)[j]);
- }
+}
PyMem_Del(*buf);
-return 0;
+total = 0;
+return total;
}
(*iov)[i].iov_base = (*buf)[i].buf;
-(*iov)[i].iov_len = (*buf)[i].len;
-}
-return 1;
+blen = (*buf)[i].len;
+(*iov)[i].iov_len = blen;
+total += blen;
+}
+return total;
}
static void
@@ -5939,14 +5945,17 @@
"offset", "count",
"headers", "trailers", "flags", NULL};
+if (!PyArg_ParseTupleAndKeywords(args, kwdict,
#ifdef __APPLE__
-if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiO&O&|OOi:sendfile",
-keywords, &out, &in, _parse_off_t, &offset, _parse_off_t, &sbytes,
+ "iiO&O&|OOi:sendfile",
+ keywords, &out, &in, _parse_off_t,
+ &offset, _parse_off_t, &sbytes,
#else
-if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiO&n|OOi:sendfile",
-keywords, &out, &in, _parse_off_t, &offset, &len,
-#endif
-&headers, &trailers, &flags))
+ "iiO&n|OOi:sendfile",
+ keywords, &out, &in,
+ _parse_off_t, &offset, &len,
+#endif
+ &headers, &trailers, &flags))
return NULL;
if (headers != NULL) {
if (!PySequence_Check(headers)) {
@@ -5954,10 +5963,15 @@
"sendfile() headers must be a sequence or None");
return NULL;
} else {
+Py_ssize_t i = 0; /* (Avoid uninitialized warning) */
sf.hdr_cnt = PySequence_Size(headers);
-if (sf.hdr_cnt > 0 && !iov_setup(&(sf.headers), &hbuf,
-headers, sf.hdr_cnt, PyBUF_SIMPLE))
+if (sf.hdr_cnt > 0 &&
+!(i = iov_setup(&(sf.headers), &hbuf,
+headers, sf.hdr_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
if (trailers != NULL) {
@@ -5966,10 +5980,15 @@
"sendfile() trailers must be a sequence or None");
return NULL;
} else {
+Py_ssize_t i = 0; /* (Avoid uninitialized warning) */
sf.trl_cnt = PySequence_Size(trailers);
-if (sf.trl_cnt > 0 && !iov_setup(&(sf.trailers), &tbuf,
-trailers, sf.trl_cnt, PyBUF_SIMPLE))
+if (sf.trl_cnt > 0 &&
+!(i = iov_setup(&(sf.trailers), &tbuf,
+trailers, sf.trl_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment:
(GMail imposes a five minute or so delay when used over SMTP, so
the timeline sometimes looks unfortunate.)
So, finally, -2.patch only fixed with 'blen' for 'len'.
Compiles and test ok.
> Giampaolo Rodola' added the comment:
>
> Please try to provide a patch which fixes (as in "makes the test pass") this
> specific issue only. As for other changes such as the code restyling you can
> open a separate ticket with another patch.
--
Added file: http://bugs.python.org/file20960/issue11351-5.patch
___
Python tracker
<http://bugs.python.org/issue11351>
___diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c
--- a/Modules/posixmodule.c
+++ b/Modules/posixmodule.c
@@ -5867,36 +5867,42 @@
#ifdef HAVE_SENDFILE
#if defined(__FreeBSD__) || defined(__DragonFly__) || defined(__APPLE__)
-static int
+static Py_ssize_t
iov_setup(struct iovec **iov, Py_buffer **buf, PyObject *seq, int cnt, int
type)
{
int i, j;
+Py_ssize_t blen, total = 0;
+
*iov = PyMem_New(struct iovec, cnt);
if (*iov == NULL) {
PyErr_NoMemory();
-return 0;
-}
+return total;
+}
+
*buf = PyMem_New(Py_buffer, cnt);
if (*buf == NULL) {
PyMem_Del(*iov);
PyErr_NoMemory();
-return 0;
+return total;
}
for (i = 0; i < cnt; i++) {
-if (PyObject_GetBuffer(PySequence_GetItem(seq, i), &(*buf)[i],
-type) == -1) {
+if (PyObject_GetBuffer(PySequence_GetItem(seq, i),
+ &(*buf)[i], type) == -1) {
PyMem_Del(*iov);
for (j = 0; j < i; j++) {
PyBuffer_Release(&(*buf)[j]);
- }
+}
PyMem_Del(*buf);
-return 0;
+total = 0;
+return total;
}
(*iov)[i].iov_base = (*buf)[i].buf;
-(*iov)[i].iov_len = (*buf)[i].len;
-}
-return 1;
+blen = (*buf)[i].len;
+(*iov)[i].iov_len = blen;
+total += blen;
+}
+return total;
}
static void
@@ -5954,10 +5960,15 @@
"sendfile() headers must be a sequence or None");
return NULL;
} else {
+Py_ssize_t i = 0; /* Avoid uninitialized warning */
sf.hdr_cnt = PySequence_Size(headers);
-if (sf.hdr_cnt > 0 && !iov_setup(&(sf.headers), &hbuf,
-headers, sf.hdr_cnt, PyBUF_SIMPLE))
+if (sf.hdr_cnt > 0 &&
+!(i = iov_setup(&(sf.headers), &hbuf,
+headers, sf.hdr_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
if (trailers != NULL) {
@@ -5966,10 +5977,15 @@
"sendfile() trailers must be a sequence or None");
return NULL;
} else {
+Py_ssize_t i = 0; /* Avoid uninitialized warning */
sf.trl_cnt = PySequence_Size(trailers);
-if (sf.trl_cnt > 0 && !iov_setup(&(sf.trailers), &tbuf,
-trailers, sf.trl_cnt, PyBUF_SIMPLE))
+if (sf.trl_cnt > 0 &&
+!(i = iov_setup(&(sf.trailers), &tbuf,
+trailers, sf.trl_cnt, PyBUF_SIMPLE)))
return NULL;
+#ifdef __APPLE__
+sbytes += i;
+#endif
}
}
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: But, dear Antoine, i don't know *any* editor that is capable to handle the following code correctly: #ifdef __APPLE__ if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiO&O&|OOi:sendfile", keywords, &out, &in, _parse_off_t, &offset, _parse_off_t, &sbytes, #else if (!PyArg_ParseTupleAndKeywords(args, kwdict, "iiO&n|OOi:sendfile", keywords, &out, &in, _parse_off_t, &offset, &len, #endif &headers, &trailers, &flags)) > Antoine Pitrou added the comment: > > Problem is, each developer has his/her own preferences and editor > settings, so if we start reformatting with each commit, the history of a > piece of code becomes very difficult to read. > > I would simply suggest that you disable these vim settings, or at least > tweak them so that they don't apply automatically when you edit existing > code. -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file20943/posixmodule.naive-apple-patch ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file20946/issue11351.patch ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Changes by Steffen Daode Nurpmeso : Removed file: http://bugs.python.org/file20949/issue11351-2.patch ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: I agree with that. > Antoine Pitrou added the comment: > If you are asking your editor to simply edit test, it should work fine. -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: > Giampaolo Rodola' added the comment: > > Please try to provide a patch which fixes (as in "makes the test pass") this > specific issue only. As for other changes such as the code restyling you can > open a separate ticket with another patch. Forza, Giampaolo. :) If you never patch file20960 in, the buildbots will fail for the time being. (And they do have to run Mac OS X already...) -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: > Giampaolo Rodola' added the comment: > > issue11351-5.patch no longer applies cleanly and I don't have an OSX box to > test against at the moment. > Could you please update your patch? If you tell me it's gonna fix the test > I'm going to commit it assuming everything is fine. I'm at e79364a6bed8/[svn r88726]. issue11351-5.patch applies and the test is ok, at least in respect to sendfile: Using random seed 1578968 [1/1] test_os /Users/steffen/arena/code.extern.repos/py3k.hg/Lib/unittest/case.py:387: ResourceWarning: unclosed file <_io.BufferedReader name='@test_59563_tmp2'> function() 1 test OK. -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment:
It's ok, but for the statistics:
16:59 ~/arena/code.extern.repos/py3k.hg $ hg identify
e79364a6bed8 (py3k) tip
16:59 ~/arena/code.extern.repos/py3k.hg $ hg slog -r tip
Changeset: 10021:e79364a6bed8
branch: py3k
tag: tip
user:giampaolo.rodola
date:Thu Mar 03 15:10:58 2011 +0100
summary: [svn r88726] fix attribute error
16:59 ~/arena/code.extern.repos/py3k.hg $ sh .hg/steffen/buildit.sh
./configure --prefix=$HOME/usr/opt/py3k MACOSX_DEPLOYMENT_TARGET=10.6
make all
17:04 ~/arena/code.extern.repos/py3k.hg $ hg patch --no-commit --strip 1
.hg/steffen/issue11351-5.patch
applying .hg/steffen/issue11351-5.patch
17:04 ~/arena/code.extern.repos/py3k.hg $ make all
/usr/bin/gcc -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -O2 -O2 -I. -IInclude -I./Include
-DPy_BUILD_CORE -c ./Modules/posixmodule.c -o
Modules/posixmodule.o
/usr/bin/gcc
...
17:05 ~/arena/code.extern.repos/py3k.hg $ ./python.exe -E -Wd -m test.test_os
-r -w -uall
test_access (__main__.FileTests) ... ok
test_closerange (__main__.FileTests) ... ok
test_read (__main__.FileTests) ... ok
test_rename (__main__.FileTests) ... ok
test_write (__main__.FileTests) ... ok
test_stat_attributes (__main__.StatAttributeTests) ... ok
test_stat_attributes_bytes (__main__.StatAttributeTests) ... ok
test_statvfs_attributes (__main__.StatAttributeTests) ... ok
test_utime_dir (__main__.StatAttributeTests) ... ok
test___repr__ (__main__.EnvironTests)
Check that the repr() of os.environ looks like environ({...}). ... ok
test_bool (__main__.EnvironTests) ... ok
test_constructor (__main__.EnvironTests) ... ok
test_environb (__main__.EnvironTests) ... ok
test_get (__main__.EnvironTests) ... ok
test_get_exec_path (__main__.EnvironTests) ... ok
test_getitem (__main__.EnvironTests) ... ok
test_items (__main__.EnvironTests) ... ok
test_keys (__main__.EnvironTests) ... ok
test_keyvalue_types (__main__.EnvironTests) ... ok
test_len (__main__.EnvironTests) ... ok
test_os_popen_iter (__main__.EnvironTests) ... ok
test_pop (__main__.EnvironTests) ... ok
test_popitem (__main__.EnvironTests) ... ok
test_read (__main__.EnvironTests) ... ok
test_setdefault (__main__.EnvironTests) ... ok
test_update (__main__.EnvironTests) ... ok
test_update2 (__main__.EnvironTests) ... ok
test_values (__main__.EnvironTests) ... ok
test_write (__main__.EnvironTests) ... ok
test_traversal (__main__.WalkTests) ... ok
test_exist_ok_existing_directory (__main__.MakedirTests) ... ok
test_exist_ok_existing_regular_file (__main__.MakedirTests) ... ok
test_makedir (__main__.MakedirTests) ... ok
test_devnull (__main__.DevNullTests) ... ok
test_urandom (__main__.URandomTests) ... ok
test_execvpe_with_bad_arglist (__main__.ExecTests) ... ok
test_execvpe_with_bad_program (__main__.ExecTests) ... ok
test_internal_execvpe_str (__main__.ExecTests) ... ok
test_closerange (__main__.TestInvalidFD) ... ok
test_dup (__main__.TestInvalidFD) ... ok
test_dup2 (__main__.TestInvalidFD) ... ok
test_fchdir (__main__.TestInvalidFD) ... ok
test_fchmod (__main__.TestInvalidFD) ... ok
test_fchown (__main__.TestInvalidFD) ... ok
test_fdatasync (__main__.TestInvalidFD) ... ok
test_fdopen (__main__.TestInvalidFD) ... ok
test_fpathconf (__main__.TestInvalidFD) ... ok
test_fstat (__main__.TestInvalidFD) ... ok
test_fstatvfs (__main__.TestInvalidFD) ... ok
test_fsync (__main__.TestInvalidFD) ... ok
test_ftruncate (__main__.TestInvalidFD) ... ok
test_isatty (__main__.TestInvalidFD) ... ok
test_lseek (__main__.TestInvalidFD) ... ok
test_read (__main__.TestInvalidFD) ... ok
test_tcgetpgrp (__main__.TestInvalidFD) ... ok
test_tcsetpgrpt (__main__.TestInvalidFD) ... ok
test_ttyname (__main__.TestInvalidFD) ... ok
test_write (__main__.TestInvalidFD) ... ok
test_setegid (__main__.PosixUidGidTests) ... ok
test_seteuid (__main__.PosixUidGidTests) ... ok
test_setgid (__main__.PosixUidGidTests) ... ok
test_setregid (__main__.PosixUidGidTests) ... ok
test_setregid_neg1 (__main__.PosixUidGidTests) ... ok
test_setreuid (__main__.PosixUidGidTests) ... ok
test_setreuid_neg1 (__main__.PosixUidGidTests) ... ok
test_setuid (__main__.PosixUidGidTests) ... ok
test_listdir (__main__.Pep383Tests) ... ok
test_open (__main__.Pep383Tests) ... ok
test_stat (__main__.Pep383Tests) ... ok
test_CTRL_BREAK_EVENT (__main__.Win32KillTests) ... skipped 'Win32 specific
tests'
test_CTRL_C_EVENT (__main__.Win32KillTests) ... skipped 'Win32 specific tests'
test_kill_int (__main__.Win32KillTests) ... skipped 'Win32 specific tests'
test_kill_sigterm (__main__.Win32KillTests) ... skipped 'Win32 specific tests'
test_directory_link (__main__.Win32SymlinkTests) ... skipped 'Win32 specific
tests'
test_file_link (__main__.Win32SymlinkTests) ... skipped 'Win32 specific tests'
test_isdir_on_directory_link_to_missing_target (__main__.Win32SymlinkTests) ...
skipped 'Win32 specific tests'
test_remove
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: > Giampaolo Rodola' added the comment: > Let's see how the buildbot goes. Italian Espressi are the best! (Our local classical radio station plays Respighi's "Pini di Roma" almost every day ...) -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Steffen Daode Nurpmeso added the comment: Buildbot ok, i hope it's ok for you all that i close this heavy-load issue. More fun further on up the road ... from a non-MQ user. -- ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11351] Mac OS X os.sendfile()
Changes by Steffen Daode Nurpmeso : -- status: open -> closed ___ Python tracker <http://bugs.python.org/issue11351> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11401] email.header error during .flatten()
New submission from Steffen Daode Nurpmeso :
Hello, David, the error from Issue 6 occurred again,
and it still can be healed with the patch file20675.
(I'm opening a new issue because 6 turned over to
the mailbox fix.
My repo and installation is at:
21:34 ~/arena/code.extern.repos/py3k.hg $ hg identify
ddd8bf84e27d+ (py3k) tip
21:34 ~/arena/code.extern.repos/py3k.hg $ hg slog -r tip
changeset: 10028:ddd8bf84e27d
branch: py3k
tag: tip
user:eli.bendersky
date:Fri Mar 04 11:38:14 2011 +0100
summary: [svn r88742]
Here is the dump:
| Opening mbox mailbox
/private/var/folders/Da/DaZX3-k5G8a57zw6MSmjJTM/-Tmp-/mail/test
David: <->
PANIC: [box] test: message-add failed.
Traceback (most recent call last):
File "/Users/steffen/usr/bin/s-postman.py", line 1093, in save_ticket
mb.add(ticket.message())
...
try:
print('David:', type(mb), ' <-> ', type(ticket.message()))
mb.add(ticket.message())
mb.flush()
except Exception as e:
efun('[box] ', self.id, ': message-add failed.\n', extb=E())
finally:
mb.unlock()
...
File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 595, in
add
self._toc[self._next_key] = self._append_message(message)
File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 733, in
_append_message
offsets = self._install_message(message)
File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 805, in
_install_message
self._dump_message(message, self._file, self._mangle_from_)
File "/Users/steffen/usr/opt/py3k/lib/python3.3/mailbox.py", line 215, in
_dump_message
gen.flatten(message)
File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line
88, in flatten
self._write(msg)
File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line
141, in _write
self._write_headers(msg)
File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/generator.py", line
373, in _write_headers
self.write(header.encode(linesep=self._NL)+self._NL)
File "/Users/steffen/usr/opt/py3k/lib/python3.3/email/header.py", line 317,
in encode
formatter.feed(lines[0], charset)
Exception: IndexError: list index out of range
In the meanwhile the thing has dispatched tens of thousands
of mails, and no error did ever occur (reading from MBox,
storing as Maildir). This happened when saving the first
message read from a Maildir mailbox to a MBox one.
I think i'll gonna do some round-trip tests tomorrow and
monday evening+. Anyway, patching in file20675 heals the
problem and the thing finishs gracefully.
--
components: Library (Lib)
messages: 130072
nosy: r.david.murray, sdaoden
priority: normal
severity: normal
status: open
title: email.header error during .flatten()
type: behavior
versions: Python 3.3
___
Python tracker
<http://bugs.python.org/issue11401>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11401] email.header error during .flatten()
Steffen Daode Nurpmeso added the comment:
> Changes by R. David Murray :
> assignee: -> r.david.murray
David, i'm sending you data to reproduce the error on your local
machine!
Please create a test directory and place the files therein
(an output.mbox will be created). Just do
mkdir -p issue_11401/input.mdir/{cur,new,tmp}
mv issue_11401_mail.txt "issue_11401/input.mdir/cur/test:2,S"
mv issue_11401_runner.py issue_11401/
cd issue_11401
python3 issue_11401_runner.py
(I'm looking forward for a glass of red wine, just in case we'll
ever meet.-)
--
Added file: http://bugs.python.org/file21005/issue_11401_mail.txt
Added file: http://bugs.python.org/file21006/issue_11401_runner.py
___
Python tracker
<http://bugs.python.org/issue11401>
___Return-Path:
X-Original-To: steffen
Delivered-To: steffen@sherwood.local
Received: by sherwood.local (Postfix, from userid 502)
id 0302C6F8D01; Wed, 5 Jan 2011 16:07:15 +0100 (CET)
MIME-Version: 1.0
Received: from gmail-pop.l.google.com [74.125.79.109]
by sherwood.local with POP3 (fetchmail-6.3.11)
for (single-drop); Wed, 05 Jan 2011 16:07:15 +0100
(CET)
Received: by 10.216.183.130; Mon, 25 Oct 2009 05:50:04 -0700 (PDT)
Date: Mon, 25 Oct 2009 05:50:04 -0700
Subject: Import your contacts and old email
From: Google Mail Team
To: Steffen Daode Nurpmeso
Content-Type: multipart/alternative; boundary=0016e6db7cbd62d15c0493706eea
Content-Length: 2490
Lines: 70
Status: RO
X-Status:
X-Keywords:
X-UID: 2
--0016e6db7cbd62d15c0493706eea
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable
You can import your contacts and mail from Yahoo!, Hotmail, AOL, and many
other web mail or POP accounts. If you want, we'll even keep importing your
mail for the next 30 days.
Import contacts and mail
=BB<https://mail.google.com/mail/#settings/accounts>
We know it can be a pain to switch email accounts, and we hope this makes
the transition to Google Mail a bit easier.
- The Google Mail Team
Please note that importing is not available if you're using Internet
Explorer 6.0. To take advantage of the latest Google Mail features,
please upgrade
to a fully supported
browser<http://mail.google.com/support/bin/answer.py?answer=3D6557&hl=3Den&=
utm_source=3Dwel-eml&utm_medium=3Deml&utm_campaign=3Den>
.
--0016e6db7cbd62d15c0493706eea
Content-Type: text/html; charset=ISO-8859-1
You can import your contacts and mail from Yahoo!, Hotmail, AOL, and many
other web mail or POP accounts. If you want, we'll even keep importing your
mail for the next 30 days.
https://mail.google.com/mail/images/welcome-button-background.png";
style="background-color: #ddd; background-repeat: repeat-x"
>https://mail.google.com/mail/#settings/accounts";
style="font-weight: bold; color: #000; text-decoration: none; display:
block; padding: 0.5em 1em"
>Import contacts and mail »
We know it can be a pain to switch email accounts, and we hope this makes
the transition to Google Mail a bit easier.
- The Google Mail Team
Please note that importing is not available if
you're using Internet Explorer 6.0. To take advantage of the latest Google Mail
features, please
http://mail.google.com/support/bin/answer.py?answer=6557&hl=en&utm_source=wel-eml&utm_medium=eml&utm_campaign=en";>
upgrade to a fully supported browser.
--0016e6db7cbd62d15c0493706eea--
import sys
import mailbox
VERBOSE = 1
def log(*args):
if not VERBOSE:
return
print('| ' + ''.join(str(i) for i in args), file=sys.stderr)
class MailboxHelper:
@staticmethod
def open(path, type='mbox', create=False):
log('Opening ', type, ' mailbox ', path)
if type == 'mbox':
mb = mailbox.mbox
elif type == 'maildir':
mb = mailbox.Maildir
#try:
mb = mb(path, create=create)
#except Exception:
# FIXME throws IOError (Errno 21) if dispatch specified a dir but
# dod not set a dir-based type..
#panic('Failed to open ', path, extb=E())
return mb
msg = None
inmb = MailboxHelper.open('input.mdir', type='maildir', create=False)
try:
log('* Box has ', len(inmb),
' messages, filtering through rulesets')
cnt = 0
for k in inmb.iterkeys():
cnt += 1
log('({!s:>04})'.format(cnt))
msg = inmb.get_message(k)
if cnt == 1:
break
except Exception as e:
error('failed to handle box ', b, extb=E())
outmb = MailboxHelper.open('output.mbox', create=True)
try:
outmb.add(msg)
outmb.flush()
except Exception as e:
log(': message-add failed.\n')
raise
finally:
outmb.unlock()
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11401] email.header error during .flatten()
Changes by Steffen Daode Nurpmeso : -- versions: +Python 3.2 ___ Python tracker <http://bugs.python.org/issue11401> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11401] email.header error during .flatten()
Steffen Daode Nurpmeso added the comment: David, it seems my patch http://bugs.python.org/file20675/email_header.patch is a real requirement for EMAIL 5.1 code. This is because BytesGenerator._write_headers() creates a Header instance in 'else:' (the other branches are not entered at all for the test mail), passing 'v' as the 's=' arg of Header ctor. It follows that Header._chunks[] is never the empty list. 'v' however may be the empty string for header fields like 'X-Status:', 'X-Keywords:' and maybe more, resulting in string.splitlines() trying to break up the empty string. I don't understand why i could pass thousands of mails, including the test mail, dozens of times through my thing without showing this up? Does the mailbox.mbox message subclass adjust such things? Bye. -- ___ Python tracker <http://bugs.python.org/issue11401> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11452] Test suite not symlink-install clean
New submission from Steffen Daode Nurpmeso :
I'm now using a completely brain-damaged way to have two 3.x python(1)s
installed: symlinks; i.e. 3.2rc2 is ~/usr/opt/.py-3.2rc2 and tip is
~/usr/opt/.cpython. Whenever i need to test the thing i'm working on
in my free time with the old version i'm temporarily replacing a
~/usr/opt/py3k symlink (which points to .cpython otherwise).
(I'm doing this because only py3k is in $PATH, $LD_LIBRARY_PATH etc.,
but like i said...)
I don't know if you consider this to be misbehaviour, but a
moment ago i've run the test suite on an installed python(1) tip
and got these symlink-path is not --prefix-path related errors,
just if anyone is concerned (stripping output a bit):
==
FAIL: test_getsourcefile (test.test_inspect.TestRetrievingSourceCode)
--
Traceback (most recent call last):
File "/Users/steffen/usr/opt/.cpython/lib/python3.3/test/test_inspect.py",
line 279, in test_getsourcefile
self.assertEqual(normcase(inspect.getsourcefile(mod.spam)), modfile)
AssertionError:
'/Users/steffen/usr/opt/.cpython/lib/python3.3/test/inspect_fodder.py' !=
'/Users/steffen/usr/opt/py3k/lib/python3.3/test/inspect_fodder.py'
- /Users/steffen/usr/opt/.cpython/lib/python3.3/test/inspect_fodder.py
?--
+ /Users/steffen/usr/opt/py3k/lib/python3.3/test/inspect_fodder.py
? ^^
==
FAIL: test_findsource_code_in_linecache (test.test_inspect.TestBuggyCases)
--
Traceback (most recent call last):
File "/Users/steffen/usr/opt/.cpython/lib/python3.3/test/test_inspect.py",
line 387, in test_findsource_code_in_linecache
self.assertRaises(IOError, inspect.findsource, co)
AssertionError: IOError not raised by findsource
==
FAIL: test_stack (test.test_inspect.TestInterpreterStack)
--
Traceback (most recent call last):
File "/Users/steffen/usr/opt/.cpython/lib/python3.3/test/test_inspect.py",
line 163, in test_stack
(modfile, 16, 'eggs', ['st = inspect.stack()\n'], 0))
AssertionError: Tuples differ: ('/Users/steffen/usr/opt/.cpyt... !=
('/Users/steffen/usr/opt/py3k/...
First differing element 0:
/Users/steffen/usr/opt/.cpython/lib/python3.3/test/inspect_fodder.py
/Users/steffen/usr/opt/py3k/lib/python3.3/test/inspect_fodder.py
- ('/Users/steffen/usr/opt/.cpython/lib/python3.3/test/inspect_fodder.py',
? --
+ ('/Users/steffen/usr/opt/py3k/lib/python3.3/test/inspect_fodder.py',
?^^
==
FAIL: test_trace (test.test_inspect.TestInterpreterStack)
--
Traceback (most recent call last):
File "/Users/steffen/usr/opt/.cpython/lib/python3.3/test/test_inspect.py",
line 174, in test_trace
(modfile, 43, 'argue', ['spam(a, b, c)\n'], 0))
AssertionError: Tuples differ: ('/Users/steffen/usr/opt/.cpyt... !=
('/Users/steffen/usr/opt/py3k/...
First differing element 0:
/Users/steffen/usr/opt/.cpython/lib/python3.3/test/inspect_fodder.py
/Users/steffen/usr/opt/py3k/lib/python3.3/test/inspect_fodder.py
- ('/Users/steffen/usr/opt/.cpython/lib/python3.3/test/inspect_fodder.py',
? --
+ ('/Users/steffen/usr/opt/py3k/lib/python3.3/test/inspect_fodder.py',
?^^
...
==
FAIL: test_loop_caller_importing (test.test_trace.TestCallers)
--
Traceback (most recent call last):
File "/Users/steffen/usr/opt/py3k/lib/python3.3/test/test_trace.py", line
286, in test_loop_caller_importing
self.assertEqual(self.tracer.results().callers, expected)
AssertionError: {(('/Users/steffen/usr/opt/.cpython/lib/python3.3/trace.py',
'trace', 'Trace.run [truncated]... !=
{(('/Users/steffen/usr/opt/py3k/lib/python3.3/test/test_trace.py',
'test_trace', [truncated]...
--
components: Tests
messages: 130452
nosy: sdaoden
priority: normal
severity: normal
status: open
title: Test suite not symlink-install clean
type: behavior
versions: Python 3.3
___
Python tracker
<http://bugs.python.org/issue11452>
___
___
Python-bugs-list mailing list
Unsubscribe:
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11452] Test suite not symlink-install clean
Steffen Daode Nurpmeso added the comment: This sounds even more silly now, but i can't reproduce the problem any more! I could do so easily this noon, before i've opened this issue! You *do* see the tracebacks! I wanted to (try to) write a patch (use os.path.realpath()), but even with drop of __pycache__'s, complete recompiles and reinstallations: i can't reproduce these faults. I've neither updated my repo, nor have i done anything else. The options are the same, too (python3 -E -Wd -m test -r -w). I leave this issue open until tomorrow noon, maybe someone with glue reads it and has one or two minutes to give me a hint on what is happening here. (Thank you!) I'll close it, then?! -- ___ Python tracker <http://bugs.python.org/issue11452> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
[issue11452] Test suite not symlink-install clean
Steffen Daode Nurpmeso added the comment: So, nobody. I still can't find a way to reproduce the errors, so. I also get occasional errors when building _multiprocessor, which disappear when (removing _multiprocessor_failed.so and) simply running make all once again. I thought about providing the patch anyway, but it is maybe better to simply close this issue (in the meanwhile). -- resolution: -> invalid status: open -> closed ___ Python tracker <http://bugs.python.org/issue11452> ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
