Re: [Python-Dev] frozendict (was: Possible patch for functools partial...)
On Sun, 16 May 2010 11:55:14 +1000 Yaniv Aknin wrote: > > Is the whole notion of a frozendict > worthy, given this limitation? I don't think so. Of course we how have a more general problem: - if we choose to implement only equality testing for partials and leave hashing as is, then hashing isn't consistent with equality anymore -- which is unacceptable - if we choose to implement only equality testing for partials and make them unhashable, we are breaking programs which store partials in a set or a dict So we are left with the following choice: - implement hashing consistently with equality testing, even though the keyword arguments can be changed later. I think this is acceptable from a practicality standpoint - abandon the whole idea of improving equality testing Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] frozendict (was: Possible patch for functools partial...)
Antoine Pitrou wrote: > On Sun, 16 May 2010 11:55:14 +1000 > Yaniv Aknin wrote: >> Is the whole notion of a frozendict >> worthy, given this limitation? > > I don't think so. Of course we how have a more general problem: > - if we choose to implement only equality testing for partials and > leave hashing as is, then hashing isn't consistent with equality > anymore -- which is unacceptable > - if we choose to implement only equality testing for partials and make > them unhashable, we are breaking programs which store partials in a > set or a dict > > So we are left with the following choice: > - implement hashing consistently with equality testing, even though the > keyword arguments can be changed later. I think this is acceptable > from a practicality standpoint Another option is to compare the keywords by identity rather than value. This is likely to be more correct anyway, since items may compare equal without giving the same result for all functions: >>> dict(a=1) == dict(a=1.0) True >>> partial(type, 1)() >>> partial(type, 1.0)() Once you're comparing the supplied arguments by identity rather than value, the question of hashability goes away. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot svn exceptions again...
Martin v. Löwis wrote: > Bill Janssen wrote: > > parc-leopard-1 (and most of the other builders) are failing the svn > > checkout with the following error: > > > > svn: PROPFIND of '/projects/python/trunk': Could not resolve hostname > > `svn.python.org': Temporary failure in name resolution > > (http://svn.python.org) > > > > When I log into that machine as "buildbot" and do the svn checkout > > manually using the following command > > > > /usr/bin/svn checkout --non-interactive --no-auth-cache --revision > > HEAD http://svn.python.org/projects/python/trunk build > > > > it works fine. > > That's a common OSX problem/bug. Processes occasionally lose the ability > to resolve host names. A workaround is to just put them in the /etc/hosts file. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot svn exceptions again...
Martin v. Löwis wrote: > That's a common OSX problem/bug. Processes occasionally lose the ability > to resolve host names. Various theories float around what's causing this > (most commonly, people expect that a "controlling terminal" must be > present); my theory is this: > > There is a Mach port in the system that lets a process talk to the local > resolver (lookupd, or whatever Apple's naming infrastructure is called > today). For some reason, this port gets closed in a "background" > process, perhaps because the parent process closed it when you logged > out. Or the Kerberos credentials expired, or some such... > > In short, if some OS X expert could shed some light on this, it would be > appreciated. It would then be good to add this to the wiki. See also This is as close to an explanation as I've been able to get from Apple: http://lists.apple.com/archives/Darwin-kernel/2009/Apr/msg00016.html > > http://buildbot.net/trac/wiki/UsingLaunchd > http://developer.apple.com/mac/library/technotes/tn2005/tn2083.html > > Regards, > Martin > > ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] buildbot svn exceptions again...
> A workaround is to just put them in the /etc/hosts file. That doesn't really help: the test suite also relies on a number of host names. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] OS X buildbots and launchd
I can find no evidence that the buildbot installation process given on the wiki will cause the buildbot slave to be restarted after a reboot of the machine. To accomplish this, you should also undertake the work described in http://buildbot.net/trac/wiki/UsingLaunchd On my Leopard slave, I created the file /Library/LaunchAgents/org.python.buildbot.slave.plist, and put in it this XML: http://www.apple.com/DTDs/PropertyList-1.0.dtd";> StandardOutPath twistd.log StandardErrorPath twistd-err.log EnvironmentVariables PATH /usr/local/bin:/sbin:/usr/sbin:/bin:/usr/bin GroupName daemon KeepAlive SuccessfulExit Label org.python.buildbot.slave ProgramArguments /usr/bin/twistd -no -y /Users/buildbot/buildarea/buildbot.tac RunAtLoad UserName buildbot WorkingDirectory /Users/buildbot/buildarea Note that I am using the system Python 2.5, and that I have configured the buildbot slave machine to login the "buildbot" account automatically upon boot, which login will cause this daemon to start, and will provide access to the GUI context for testing tk. I've configured it with a locking screensaver that starts immediately; let's see if that works OK. I've also put "dinsdale.python.org" and "svn.python.org" in /etc/hosts. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] what ports are needed for a buildbot?
I'm running my buildbots in a captive firewalled subnet, and I have to enumerate all the ports that the buildbot is allowed to open to the outside world. Right now, I'm getting this error: ERROR: test_connect (test.test_smtpnet.SmtpSSLTest) -- Traceback (most recent call last): File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/test/test_smtpnet.py", line 15, in test_connect server = smtplib.SMTP_SSL(self.testServer, self.remotePort) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 757, in __init__ SMTP.__init__(self, host, port, local_hostname, timeout) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 239, in __init__ (code, msg) = self.connect(host, port) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 295, in connect self.sock = self._get_socket(host, port, self.timeout) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/smtplib.py", line 762, in _get_socket new_socket = socket.create_connection((host, port), timeout) File "/Users/buildbot/buildarea/3.x.parc-leopard-1/build/Lib/socket.py", line 321, in create_connection raise error(msg) socket.error: [Errno 61] Connection refused This make me think that the list of ports enumerated at http://wiki.python.org/moin/BuildBot is incomplete. We need a complete list. If you know of additional ports, please add them to the wiki page. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Fixing the GIL (with a BFS scheduler)
Hi all, Here is a second (last?) attempt at getting traction on fixing the GIL (is it broken?) with BFS (WTF?). So don't be shy (don't be too rude either) since ignoring counts as down voting. Relevant Python issue: http://bugs.python.org/issue7946 *Bottom line first* I submitted an implementation of BFS ( http://ck.kolivas.org/patches/bfs/sched-BFS.txt) as a patch to the GIL, which to the extent I have tested it, behaves nicely on Windows XP, Windows 7, GNU/Linux with either CFS or O(1) schedulers, 1/2/4 cores, laptop, desktop and VirtualBox VM guest (some data below). The patch is still work in progress and requires work in terms of style, moving code where it belongs, test code, etc... nevertheless, Python core developers recommended I already (re)post to python-dev for discussion. *So is the GIL broken?* There seems to be some disagreement on that question among Python core developers (unless you all agree it is not broken :) ). Some developers maintain the effects described by David Beazley do not affect real world systems. Even I took the role of a devil's advocate in a previous discussion, but in fact I think that Python, being a general purpose language, is similar to the OS in that regard. It is used across many application domains, platforms, and development paradigms, just as OS schedulers are, and therefore accepting thread scheduling with such properties as a fact of life is not a good idea. I was first bitten by the original GIL last year while testing a system, and found David's research while looking for answers, and later had to work around that problem in another system. Here are other real world cases: 1) Zope people hit this back in 2002 and documented the problem with interesting insight: http://www.zope.org/Members/glpb/solaris/multiproc "I have directly observed a 30% penalty under MP constraints when the sys.setcheckinterval value was too low (and there was too much GIL thrashing)." http://www.zope.org/Members/glpb/solaris/report_ps "A machine that's going full-throttle isn't as bad, curiously enough -- because the other CPU's are busy doing real work, the GIL doesn't have as much opportunity to get shuffled between CPUs. On a MP box it's very important to set sys.setcheckinterval() up to a fairly large number, I recommend pystones / 50 or so." 2) Python mailing list - 2005 http://mail.python.org/pipermail/python-list/2005-August/336286.html "The app suffers from serious performance degradation (compared to pure c/C++) and high context switches that I suspect the GIL unlocking may be aggravating ?" 3) Python mailing list - 2008 http://mail.python.org/pipermail/python-list/2008-June/1143217.html "When I start the server, it sometimes eats up 100% of the CPU for a good minute or so... though none of the threads are CPU-intensive" 4) Twisted http://twistedmatrix.com/pipermail/twisted-python/2005-July/011048.html "When I run a CPU intensive method via threads.deferToThread it takes all the CPU away and renders the twisted process unresponsive." Admittedly, it is not easy to dig reports up in Google. Finally, I think David explained the relevance of this problem quite nicely: http://mail.python.org/pipermail/python-dev/2010-March/098416.html *What about the new GIL?* There is no real world experience with the new GIL since it is under development. What we have is David's analysis and a few benchmarks from the bug report. *Evolving the GIL into a scheduler* The problem addressed by the GIL has always been *scheduling* threads to the interpreter, not just controlling access to it. The patches by Antoine and David essentially evolve the GIL into a scheduler, however both cause thread starvation or high rate of context switching in some scenarios (see data below). *BFS* Enter BFS, a new scheduler designed by Con Kolivas, a Linux kernel hacker who is an expert in this field: http://ck.kolivas.org/patches/bfs/sched-BFS.txt "The goal of the Brain Fuck Scheduler, referred to as BFS from here on, is to completely do away with the complex designs of the past for the cpu process scheduler and instead implement one that is very simple in basic design. The main focus of BFS is to achieve excellent desktop interactivity and responsiveness without heuristics and tuning knobs that are difficult to understand, impossible to model and predict the effect of, and when tuned to one workload cause massive detriment to another." I submitted an implementation of BFS (bfs.patch) which on my machines gives comparable performance to gilinter2.patch (Antoine's) and seems to schedule threads more fairly, predictably, and with lower rate of context switching (see data below). There are however, some issues in bfs.patch: 1) It works on top of the OS scheduler, which means (for all GIL patches!): a) It does not control and is not aware of information such as OS thread preemption, CPU core to run on, etc... b) There may be hard to predict interaction between BFS and the underlying
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
I'm interested in having *something*, but I'm not particularly interested in the 3.x branch. I expect it to be at least 5 years before the various Python subsystems I depend on are all ported to 3.x, and I expect it to be at least that long before I can move onto a version of OS X that ships with Python 3 as the default system Python. Right now, I'm faced with the prospect of Apple's next OS including Python 2.7, and being saddled with the current Python 2.x terrible multicore behavior for the next 5 years or so. We badly need some kind of patch for this in the 2.x branch. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X buildbots and launchd
Bill Janssen wrote: > I can find no evidence that the buildbot installation process given on > the wiki will cause the buildbot slave to be restarted after a reboot of > the machine. To accomplish this, you should also undertake the work > described in > > http://buildbot.net/trac/wiki/UsingLaunchd > > On my Leopard slave, I created the file > /Library/LaunchAgents/org.python.buildbot.slave.plist, and put in it > this XML: > > [...] > > Note that I am using the system Python 2.5, and that I have configured > the buildbot slave machine to login the "buildbot" account automatically > upon boot, which login will cause this daemon to start, and will provide > access to the GUI context for testing tk. I've configured it with a > locking screensaver that starts immediately; let's see if that works OK. This seems to work. test_tk now passes. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Bill Janssen wrote: > Right now, I'm faced with the prospect of Apple's next OS including > Python 2.7, and being saddled with the current Python 2.x terrible > multicore behavior for the next 5 years or so. We badly need some kind > of patch for this in the 2.x branch. The matter of the GIL seems far less urgent to those of us that don't see threading as a particularly good way to exploit multiple cores. Either way, with the first 2.7 release candidate out soon, it's already too late to contemplate significant changes to the GIL for that release. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Hi, Le dimanche 16 mai 2010 22:07:06, Nir Aides a écrit : > *Evolving the GIL into a scheduler* > > The problem addressed by the GIL has always been *scheduling* threads to > the interpreter, not just controlling access to it. The patches by Antoine > and David essentially evolve the GIL into a scheduler, however both cause > thread starvation or high rate of context switching in some scenarios (see > data below). I didn't followed last development around the GIL. Can you explain me why Python should have its own scheduler whereas each OS has already its own scheduler? The OS has useful informations to help scheduling that Python doesn't have. Linux and FreeBSD schedulers are faster each year since... 5 years?, especially with multiple CPU/cores. -- Victor Stinner http://www.haypocalc.com/ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Nick Coghlan wrote: > Bill Janssen wrote: > > Right now, I'm faced with the prospect of Apple's next OS including > > Python 2.7, and being saddled with the current Python 2.x terrible > > multicore behavior for the next 5 years or so. We badly need some kind > > of patch for this in the 2.x branch. > > The matter of the GIL seems far less urgent to those of us that don't > see threading as a particularly good way to exploit multiple cores. Nick, this isn't about exploiting cores. This is about Python programs that used to work fine on single-core machines suddenly becoming horrible resource hogs when moved to a more modern machine with a more modern version of Python. As far as I'm concerned, just tying all of the program's threads to a single core would be fine, though I imagine others would differ. > Either way, with the first 2.7 release candidate out soon, it's already > too late to contemplate significant changes to the GIL for that release. The release schedule, and labelling things as "release candidates" or not, are all under our control. Nothing is "too late". And there's always Python 2.8 :-). But I'd consider this a bug in the threading library, not some unmotivated blue-sky change to the GIL. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Bill Janssen wrote: > As far as I'm concerned, just tying all of > the program's threads to a single core would be fine, though I imagine > others would differ. Which can be done through the OS tools for setting an application's CPU affinity. Fixing the Python thread scheduling is only necessary if we want to be able to exploit the extra power of those cores rather than forcing reversion to single core behaviour. Note that I'm not *opposed* to fixing it, and the discussion in the tracker issue over Nir and Dave's different approaches to the problem looks interesting. > The release schedule, and labelling things as "release candidates" or > not, are all under our control. Nothing is "too late". And there's > always Python 2.8 :-) . But I'd consider this a bug in the threading > library, not some unmotivated blue-sky change to the GIL. Yes, but if we never said "too late" we'd never ship anything :) And you do have a reasonable case for considering this a bug, but it wouldn't be the first time we've escalated bug fixes to "new feature" level simply because they had a relatively high impact on core parts of the code. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Nick Coghlan wrote: > Bill Janssen wrote: > > As far as I'm concerned, just tying all of > > the program's threads to a single core would be fine, though I imagine > > others would differ. > > Which can be done through the OS tools for setting an application's CPU > affinity. Yes, as I say, if the initialization of the threading module called those tools appropriately and automatically, I'd be happy. Well, less unhappy :-). I have to admit, I don't know how to do this on OS X. What's the tool and the process, if we're not getting too far afield? Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Bill Janssen wrote: > Nick Coghlan wrote: > >> Bill Janssen wrote: >>> As far as I'm concerned, just tying all of >>> the program's threads to a single core would be fine, though I imagine >>> others would differ. >> Which can be done through the OS tools for setting an application's CPU >> affinity. > > Yes, as I say, if the initialization of the threading module called > those tools appropriately and automatically, I'd be happy. Well, less > unhappy :-). > > I have to admit, I don't know how to do this on OS X. What's the > tool and the process, if we're not getting too far afield? OSX doesn't really support thread affinity. The affinity API that they have is described on http://developer.apple.com/mac/library/releasenotes/Performance/RN-AffinityAPI/index.html You can't bind a thread to specific core with it, though, but you can requests that multiple threads all run on the same core (leaving the choice "which core" to the system). IIUC, an affinity preference does not survive exec(2), so you can't write a tool that binds all threads of its child processes to a single core (such a tool is available on many unices, though). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Martin v. Löwis wrote: > OSX doesn't really support thread affinity. The affinity API that they > have is described on > > http://developer.apple.com/mac/library/releasenotes/Performance/RN-AffinityAPI/index.html > > You can't bind a thread to specific core with it, though, but you can > requests that multiple threads all run on the same core (leaving the > choice "which core" to the system). I believe that would be sufficient to fix the problem with Python, though I wonder about the effect on JCC-generated modules like pylucene, where the threads are really Java threads as well as Python threads. So the patch to the threading code would presumably, for those OSs where the capability exists, try to put all created threads in the same affinity set. Presumably there would also be a way to clear that binding, for folks who know what they're doing. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] OS X buildbots and launchd
I've repeated this setup with OS X 10.4 (Tiger). Some changes to the XML file were necessary, because I'd installed a non-system Python, twisted, and zope.interface on this machine, and because on Tiger, launchd daemon were not allowed to set their group ID. Here's the Tiger version, which again goes in /Library/LaunchAgents/org.python.buildbot.slave.plist. Bill http://www.apple.com/DTDs/PropertyList-1.0.dtd";> StandardOutPath twistd.log StandardErrorPath twistd-err.log EnvironmentVariables PATH /Library/Frameworks/Python.framework/Versions/2.6/bin:/sbin:/usr/sbin:/bin:/usr/bin PYTHONPATH /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages KeepAlive SuccessfulExit Label org.python.buildbot.slave ProgramArguments /Library/Frameworks/Python.framework/Versions/2.6/bin/twistd -no -y /Users/buildbot/buildarea/buildbot.tac RunAtLoad UserName buildbot WorkingDirectory /Users/buildbot/buildarea ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Sun, May 16, 2010 at 22:52, Victor Stinner wrote: > I didn't followed last development around the GIL. Can you explain me why > Python should have its own scheduler whereas each OS has already its own > scheduler? Because the GIL locks and unlocks threads, in practice, it already have. But the scheduler is so simplistic it ends up fighting with the OS scheduler, and a large amount of CPU time is used up switching instead of executing. Having a proper scheduler fixes this. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
