Re: [Python-Dev] FYI: building on OS X
Richard Jones wrote: > I just built 2.5 SVN on my OS X 10.4 laptop. I ran into the missing > sys/statvfs.h problem that has been reported here in the past. To > solve the problem I needed to upgrade my XCode install from 1.1 to > 2.2.1 - the missing header files (there was actually a whole lot of > them) were then installed. > > I don't know whether this needs to be mentioned somewhere in the > build docs. I think adding a remark to the build docs is NOT the best solution. Instead, it would be good if the problem was fixed. Looking at earlier communication, it appears that configure detects the presence of fstatvfs(), and posixmodule.c infers from that that sys/statvfs.h must be present. This is apparently misleading, as you can manage to install OSX so that the header files installed don't match the system that is running. I added some additional autoconf tests in 46010 and 46011; unfortunately, I cannot test these changes as I don't have a system where the build was broken. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
Martin v. Löwis wrote:
> So if the the import lock was held during the fork call, the new thread
> will hold the import lock of the new process, and subsequent imports
> will block.
> However, then the problem is not with the execve implementation, but
> with the fact that the import lock was held when the process forked.
>
> Rotem should simply avoid to fork() in the toplevel code of a module.
That's what I originally thought, but Rotem was able to clarify for me that
the problem occurs even if the import lock is held by a *different* thread at
the time fork() is called. So a deadlock can be triggered by perfectly
legitimate code that has the misfortune to fork the process while another
thread is doing a module import.
> Unfortunately, that fix would apply to AIX only, though:
>
> void
> _PyImport_ReInitLock(void)
> {
> #ifdef _AIX
> if (import_lock != NULL)
> import_lock = PyThread_allocate_lock();
> #endif
> }
>
> If we want to support fork while an import is going on, we should
> release the import lock if it is held. Alternatively, the code
> could throw away the old import lock on all systems; that seems
> like a waste of resources to me, though. One should also reset
> import_lock_thread and import_lock_level.
Broadening the ifdef to cover all posix systems rather than just AIX might be
the right thing to do. However, the first thing would be to reproduce the
deadlock reliably.
A repeatable test case would look something like this:
class ForkingThread(Thread):
def run(self):
# create a subprocess using subprocess.Popen
# and ensure it tries to do an import operation
# (i.e. something similar to what Rotem & Yair's code is doing)
class ImportingThread(Thread):
def __init__(self):
Thread.__init__(self)
self.importing = Event()
self.cleanup = Event()
def run(self):
imp.acquire_lock()
try:
self.importing.set()
self.cleanup.wait()
finally:
imp.release_lock()
def test_forked_import_deadlock():
import_thread = ImportingThread()
fork_thread = ForkingThread()
import_thread.start()
try:
import_thread.importing.wait()
fork_thread.start() # Fork while the import thread holds the lock
fork_thread.join(10) # Give the subprocess time to finish
assert not fork_thread.isAlive() # A timeout means it deadlocked
finally:
import_thread.cleanup.set() # Release the import lock
Cheers,
Nick.
--
Nick Coghlan | [EMAIL PROTECTED] | Brisbane, Australia
---
http://www.boredomandlaziness.org
___
Python-Dev mailing list
[email protected]
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r45925 - in python/trunk: Lib/tempfile.py Lib/test/test_os.py Misc/NEWS Modules/posixmodule.c
Martin v. Löwis wrote: > M.-A. Lemburg wrote: >> Well, the strings and integers count twice: once in the module >> namespace and once in the errorcode dictionary. > > That shouldn't be the case: the strings are interned (as they > are identifier-like), so you have the very same string object > in both dictionaries. > > The numbers shouldn't be duplicated because they occur > in the co_consts array of the global code object, and because > the compiler should share them there. Hmm, you're right. >> Given that the code strings and integers are created >> twice in my version of the module, the numbers sound about >> right. > > If they are indeed created twice, something is wrong. > >> I agree that creating only one dictionary statically >> and the other mapping dynamically will already be a >> saving of 50% simply by sharing the string and integer >> objects. > > No, they should be shared already, so that shouldn't save > anything. I changed the generator to now use only one dictionary and create the module symbols at import time. Here's the new data: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND Before: lemburg 29980 0.0 0.4 19936 4620 pts/3S+ 13:27 0:00 python After the import: lemburg 29980 0.0 0.5 20756 5260 pts/3S+ 13:27 0:00 python The RSS changed by 640kB - only 60kB less than with the static approach. I also tried this on a 32-bit machine: the RSS changes by 392kB. Next to come: the C extension version... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2006) >>> Python/Zope Consulting and Support ...http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ...http://python.egenix.com/ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] MSVC 6.0 patch
Hi, This patch submitted in March allows Python trunk to be built using MSVC 6: http://sourceforge.net/tracker/index.php?func=detail&aid=1457736&group_id=5470&atid=305470 It is not my patch but I am quite interested because the code changes are also required to build with MS eMbedded Visual C++ 4.0, the Windows CE 4.x compiler. I'm guessing that the person assigned to the patch is too busy so I hope somebody else can take over? Thanks, Luke ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] total ordering.
Guido van Rossum wrote: > On 5/11/06, Vladimir 'Yu' Stepanov <[EMAIL PROTECTED]> wrote: >> If for Python-3000 similar it will be shown concerning types >> str(), int(), complex() and so on, and the type of exceptions >> will strongly vary, it will make problematic redefinition of >> behavior of function of sorting. > > Not really. We'll just document that sort() should only be used on a > list of objects that implement a total ordering. The behavior > otherwise will simply be undefined; it will raise whatever exception > is first raised by an unsupported comparison (most likely TypeError). > In practice this won't be a problem. > In my opinion for functions of comparisons there is no additional kind of exceptions. If made action is not operation of reduction of type more often exception TypeError means a mistake of programming. At creation of exception, characteristic only for comparison, it is possible to involve methods `.__r(eq|ne|le|lt|ge|gt|cmp)__()' not being afraid to hide a mistake of programming TypeError (sorry for a tautology). It will be possible it conveniently to use as exception of management by a stream, for indication of necessity to involve `.__r(eq|ne|le|lt|ge|gt|cmp)__()' a method. This kind of a class can carry out function, similarly to StopIteration for `.next()'. In case of unsuccessful comparison the user has an opportunity in function `cmp' a method .sort() simple image to define the behaviour necessary to it, including an opportunity of sorting of diverse elements. At present time similar function is carried out with exception NotImplemented. This exception is generated in a number of mathematical operations. For this reason I ask to consider an opportunity of creation of a new class. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] total ordering.
On 5/11/06, Vladimir 'Yu' Stepanov <[EMAIL PROTECTED]> wrote: > If for Python-3000 similar it will be shown concerning types > str(), int(), complex() and so on, and the type of exceptions > will strongly vary, it will make problematic redefinition of > behavior of function of sorting. I don't see what you mean by "redefinition of behavior of function of sorting". Is this something a Python programmer might want to do? Can you give an example? On 5/16/06, Vladimir 'Yu' Stepanov <[EMAIL PROTECTED]> wrote: > It will be possible it conveniently to use as exception of > management by a stream, for indication of necessity to involve > `.__r(eq|ne|le|lt|ge|gt|cmp)__()' a method. This kind of a class > can carry out function, similarly to StopIteration for `.next()'. There are no .__r(eq|ne|le|lt|ge|gt|cmp)__() methods, for a logical reason which you might enjoy deducing yourself... > At present time similar function is carried out with exception > NotImplemented. This exception is generated in a number of > mathematical operations. For this reason I ask to consider an > opportunity of creation of a new class. Can you explain this? NotImplemented isn't an exception. (NotImplementedError is, but that's something quite different.) NotImplemented has exactly one purpose in Python, as far as I can tell. What mathematical operations do you mean? -j ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] [Python-checkins] r46002 - in python/branches/release24-maint: Misc/ACKS Misc/NEWS Objects/unicodeobject.c
[M.-A. Lemburg] >>> Could you please make this fix apply only on Solaris, >>> e.g. using an #ifdef ?! [Martin v. Löwis] >> That shouldn't be done. The code, as it was before, had >> undefined behaviour in C. With the fix, it is now correct. [Marc-Andre] > I don't understand - what's undefined in: > > const char *s; > Py_UNICODE *p; > ... > *p = *(Py_UNICODE *)s; The pointer cast: A pointer to an object or incomplete type may be converted to a pointer to a different object or incomplete type. If the resulting pointer is not correctly aligned for the pointed-to type, the behavior is undefined. Since Py_UNICODE has a stricter alignment requirement than char, there's no guarantee that _the content_ of p is correctly aligned for Py_UNICODE after the cast. Indeed, that's why the code segfaulted on the Solaris box. On other architectures it may not segfault but "just" take much longer for the HW and SW to hide improperly aligned access. >> If you want to drop usage of memcpy on systems where you >> think it isn't needed, you should make a positive list of >> such systems, e.g. through an autoconf test (although such >> a test is difficult to formulate). > I don't want to drop memcpy() - just keep the existing > working code on platforms where the memcpy() is not > needed. There's no clear way I know of to guess which platforms that may be. Is it possible to fiddle _PyUnicode_DecodeUnicodeInternal's _callers_ so that the char* `s` argument passed to it is always properly aligned for Py_UNICODE? Then the pointer cast would be fine. > ... > A modern compiler should know the alignment requirements > of Py_UNICODE* on the platform and generate appropriate > code. The trend in modern compilers and architectures is to be less forgiving of standard violations, not more. > AFAICTL, only 64-bit platforms are subject to any > such problems due to their requirement to have pointers > aligned on 8-byte boundaries. It's not the alignment of the pointer but of what the pointer points _at_ that's at issue here. While the effect of the pointer cast is undefined, it's not the pointer cast that blows up. It's dereferencing the _result_ of the pointer cast that blows up: it was trying to read up a Py_UNICODE from an address that wasn't properly aligned for Py_UNICODE. That can blow up (or be very slow, or return gibberish -- it's undefined) even if Py_UNICODE has an alignment requirement of "just" 2 (which I expect was actually the case on the Solaris box). ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
Rotem Yaari wrote: >> Rotem should simply avoid to fork() in the toplevel code of a module. >> > > That's exactly the problem - the fork is not done at the module level, > and the reason for the deadlock is imports inside functions performed > by various library functions. > This can happen virtually in every possible step of the program as > long as the python builtins keep doing it. Well, Python *builtins* (in the strict sense of the word) usually never directly import anything, and most of them not even indirectly import (there are some exceptions, of course, like the __import__ builtin) So if you are certain that the fork doesn't happen *while* an import is going, then I think the specific problem must be analyzed in more detail before an attempt is made to fix it. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
Yair Chuchem wrote: > Rotem should simply avoid to fork() in the toplevel code of a module. > > > this is not what happened. > we called subprocess which itself called fork. Ok - but are you calling subprocess in the context of code that is just being imported? Or is the caller of the code that calls subprocess just being imported? I.e. if you would trace back the stack at the point of fork, would there be an import statement on any frame of the stack (no matter how deeply nested)? > as a subprocess bug this could be easily fixed by moving the import in > "os._execve". > we have made that fix locally and it fixes the problem. I can readily believe that the problem went away. However, you did not fix it - you worked around it. The import lock *ought* to be available inside the execve code, so the import *ought* to work. By moving the import outside execve, you have only hidden the problem, instead of solving it. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] pthreads, fork, import, and execvp
Nick Coghlan wrote: > Broadening the ifdef to cover all posix systems rather than just AIX > might be the right thing to do. As I said: I doubt it is the right thing to do. Instead, the lock should get released in the child process if it is held at the point of the fork. I agree that we should find a repeatable test case first. Contributions are welcome. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
