Re: [Python-Dev] Possible patch for functools partial - Interested?
On 05/14/2010 06:39 AM, Daniel Urban wrote: I've made a new patch, in which the keywords attribute is a read-only proxy of the dictionary. What about backward compatibility? This looks like an incompatible change. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible patch for functools partial - Interested?
On Mon, May 17, 2010 at 09:47, Hrvoje Niksic wrote: > On 05/14/2010 06:39 AM, Daniel Urban wrote: >> >> I've made a new patch, in which the keywords attribute is a read-only >> proxy of the dictionary. > > What about backward compatibility? This looks like an incompatible change. You're probably right. I'd be happy to make patch which doesn't replace the dict with the proxy. But that would mean, that the hash value isn't really immutable. Daniel Urban ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Mon, 17 May 2010 08:28:08 +0200 Lennart Regebro wrote: > But the scheduler is so simplistic it ends up fighting with the > OS scheduler, and a large amount of CPU time is used up switching > instead of executing. This is already fixed with py3k. Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Sun, 16 May 2010 15:13:44 PDT Bill Janssen wrote: > > So the patch to the threading code would presumably, for those OSs where > the capability exists, try to put all created threads in the same > affinity set. This is not really a good idea. There's some code which releases the GIL, precisely so that you can run several threads (computations) at once. If you aren't too concerned with interactivity, you can increase sys.setcheckinterval() to alleviate the problem on 2.x. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: > On Mon, 17 May 2010 08:28:08 +0200 > Lennart Regebro wrote: >> But the scheduler is so simplistic it ends up fighting with the >> OS scheduler, and a large amount of CPU time is used up switching >> instead of executing. > > This is already fixed with py3k. Are you referring to the "New GIL"? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Mon, 17 May 2010 14:47:25 +0200 Lennart Regebro wrote: > On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: > > On Mon, 17 May 2010 08:28:08 +0200 > > Lennart Regebro wrote: > >> But the scheduler is so simplistic it ends up fighting with the > >> OS scheduler, and a large amount of CPU time is used up switching > >> instead of executing. > > > > This is already fixed with py3k. > > Are you referring to the "New GIL"? Yes. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: > > > > So the patch to the threading code would presumably, for those OSs where > > the capability exists, try to put all created threads in the same > > affinity set. > > This is not really a good idea. Yes, fixing the GIL for multicore in 2.7 would be a better idea, IMO. But that might be too large a change. > There's some code which releases the GIL, precisely so that you can > run several threads (computations) at once. If they can get hold of the GIL in the first place! Yes, you'd want to be able to "unbind" threads if you knew what you were doing, so that they could run on other cores, and you'd want a switch to disable the affinity mechanism entirely. But I'd be happy to have things in the naive case run as well as they do on single-core machines, and let experts do optimizations. > If you aren't too concerned with interactivity, you can increase > sys.setcheckinterval() to alleviate the problem on 2.x. Unfortunately, many use cases might well be concerned with interactivity. Things like event-processing loops. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Le lundi 17 mai 2010 à 09:05 -0700, Bill Janssen a écrit : > Antoine Pitrou wrote: > > > On Sun, 16 May 2010 15:13:44 PDT > > Bill Janssen wrote: > > > > > > So the patch to the threading code would presumably, for those OSs where > > > the capability exists, try to put all created threads in the same > > > affinity set. > > > > This is not really a good idea. > > Yes, fixing the GIL for multicore in 2.7 would be a better idea, IMO. > But that might be too large a change. Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch changing fundamental threading properties is a no-no (even the processor affinity proposal). Someone had tried to backport the new GIL to 2.7 (there's a tracker issue for it, I'm too lazy to search right now), but it was concluded that compatibility requirements for Python 2.x (compatibility with various legacy threading libraries) made things too complicated and tedious. > > There's some code which releases the GIL, precisely so that you can > > run several threads (computations) at once. > > If they can get hold of the GIL in the first place! Yes, you'd want to > be able to "unbind" threads if you knew what you were doing, so that > they could run on other cores, and you'd want a switch to disable the > affinity mechanism entirely. But I'd be happy to have things in the > naive case run as well as they do on single-core machines, and let > experts do optimizations. "Letting experts do optimizations" is a regression, though, because right now you don't have to be an expert to take advantage of such optimizations (just run a separate thread doing e.g. some zlib compression). Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Antoine Pitrou wrote: > Well, 2.7rc1 is scheduled in less than three weeks now. IMO any patch > changing fundamental threading properties is a no-no (even the processor > affinity proposal). Unfortunately, our "fundamental threading properties" are broken for multicore machines. And multicore seems to be the wave of the future. Not fixing this is a big problem. It relegates the only Python which will be usable by many many people for many years, 2.x, to a toy language status on modern machines. It will have threads, but the use of them, either directly or indirectly by modules you may import, may cause unpredictable performance problems. I'd hate to let a fundamental flaw like this go through simply because someone somewhere somewhen set a completely synthetic deadline. > Someone had tried to backport the new GIL to 2.7 (there's a tracker > issue for it, I'm too lazy to search right now), but it was concluded > that compatibility requirements for Python 2.x (compatibility with > various legacy threading libraries) made things too complicated and > tedious. http://bugs.python.org/issue7753, from January. I've read through that issue (several times), and I have to say that I wind up gnashing my teeth each time. Here's a fix that's rejected because it "only" supports NT and POSIX threads. What percentage of Python use cases do those two threading systems cover? Do we know? If by "compatibility requirements" you mean that new releases of Python should run on antique systems just as the older releases did, that's satisfied by the issue's current state, you're right. On the other hand, to me that seems an odd goal to prioritize. After all, the older releases are still there, for that antique system. Nor do I see an answer to Russ' final question: ``What if, as you proposed earlier, the patch were to leave the old behavior if the threading model on the given platform were not supported?'' > > > There's some code which releanses the GIL, precisely so that you can > > > run several threads (computations) at once. > > > > If they can get hold of the GIL in the first place! Yes, you'd want to > > be able to "unbind" threads if you knew what you were doing, so that > > they could run on other cores, and you'd want a switch to disable the > > affinity mechanism entirely. But I'd be happy to have things in the > > naive case run as well as they do on single-core machines, and let > > experts do optimizations. > > "Letting experts do optimizations" is a regression, though, because > right now you don't have to be an expert to take advantage of such > optimizations (just run a separate thread doing e.g. some zlib > compression). If threading performance wasn't broken on multicore, I'd agree with you. But right now, *everyone* has to be an expert just to use Python 2.x effectively on modern multicore hardware -- you have to find the right patch in issue 7753, apply it to the sources, build a custom python, and use it. Whether or not use explicitly know you are using threads (because some other package may be using them under the covers). Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Mon, 17 May 2010 10:11:03 PDT Bill Janssen wrote: > > I'd hate to let a fundamental flaw like this go through simply because > someone somewhere somewhen set a completely synthetic deadline. [...] > I've read through that issue (several times), and I have to say that I > wind up gnashing my teeth each time. Here's a fix that's rejected > because it "only" supports NT and POSIX threads. What percentage of > Python use cases do those two threading systems cover? Do we know? Well, if instead of gnashing your teeth, you had contributed to the issue, perhaps a patch would have been committed by now (or perhaps not, but who knows?). If you stay silent, you cannot assume that someone else will stand up for *your* opinion (and the fact that nobody did could indicate that not many people care about the issue, actually). > But right now, *everyone* has to be an expert just to use Python 2.x > effectively on modern multicore hardware Python works reasonably well on multicore hardware, especially if you don't run spinning loops and if you're not on Mac OS X. It may lose *at most* 10-20% performance compared to a single-core run but that's hardly the end of the world. And some workloads won't suffer any degradation. Besides, today's multicore CPUs have far better single-threaded performance than yesteryear's single-core CPUs, which makes the performance regression issue more theoretical than practical. In real life, you have very little risk of witnessing a performance regression when switching your Python from a single-core to a multicore machine. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Bill Janssen wrote: > use it. Whether or not use explicitly know you are using threads > (because some other package may be using them under the covers). Of course, I meant to say, "Whether or not *youse* explicitly know you are using threads (because some other package may be using them under the covers)." :-). Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Antoine Pitrou wrote: > Well, if instead of gnashing your teeth, you had contributed to the > issue, perhaps a patch would have been committed by now (or perhaps > not, but who knows?). If you stay silent, you cannot assume that > someone else will stand up for *your* opinion (and the fact that nobody > did could indicate that not many people care about the issue, actually). Unfortunately, I somehow did not even *know* about the issue until February, after the issue had been closed. What I did know was that some of our big complicated Python multi-threaded daemons had shown puzzling resource hogging when moved from small Macs to large 8-core machines with hardware RAID and lots of memory. But, simpleton that I am, I'd presumed that threading in Python wasn't broken, and was looking elsewhere for the cause. > Python works reasonably well on multicore hardware, especially if you > don't run spinning loops and if you're not on Mac OS X. I'm not sure what you mean by "spinning loops". But I *am* on Mac OS X, along with an increasing percentage of the world. And I'm dismayed that there's no momentum to fix this problem. Not a good sign. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Mon, 17 May 2010 11:15:49 PDT Bill Janssen wrote: > > What I did know was that > some of our big complicated Python multi-threaded daemons had shown > puzzling resource hogging when moved from small Macs to large 8-core > machines with hardware RAID and lots of memory. Could you give detailed information about this? Since you're talking about a "big complicated Python multi-threaded daemon", I presume you can't port it to Python 3 very quickly, but it would be nice to know if the problem disappears with 3.2. > I'm not sure what you mean by "spinning loops". It was an allusion to Dave Beazley's first benchmarks, which merely ran a spinning loop over several threads, and showed catastrophic degradation under OS X. > But I *am* on Mac OS X, along with an increasing percentage of the > world. And I'm dismayed that there's no momentum to fix this problem. There /has/ been momentum in fixing it. In py3k. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
2010/5/16 Nir Aides > > *What can be done with it?* > > Here are some options: > 1) Abandon it - no one is interested, yawn. > 2) Take ideas and workarounds from its code and apply to other patches. > 3) Include it in the interpreter as an auxiliary (turn on with a runtime > switch) scheduler. > 4) Adopt it as the Python scheduler. > > > *Opinion?* > > I would like to have the possibility to "./configure --without-broken-GIL" or "./configure --with-bfs-scheduler" in Python 2.7 like we "./configure --with-computed-gotos" in 3.2. It will let the opportunity for the experienced user (or the distribution packager) to enable the threading optimizations on its platform, while it preserves the current behaviour by default. It will give more chance that people test this experimental configure option and report any issue they find, so it can be fixed and improved in the next bugfix version (2.7.1). Since the 2.7 branch will have long term support, it makes sense to support multi-core platforms. And more users of the fixed GIL means more bugfixes (even for 3.x branch). -- Florent ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Antoine Pitrou wrote: > On Mon, 17 May 2010 11:15:49 PDT > Bill Janssen wrote: > > > > What I did know was that > > some of our big complicated Python multi-threaded daemons had shown > > puzzling resource hogging when moved from small Macs to large 8-core > > machines with hardware RAID and lots of memory. > > Could you give detailed information about this? Probably not detailed enough. IP issues. It's a version of UpLib. > Since you're talking about a "big complicated Python multi-threaded > daemon", I presume you can't port it to Python 3 very quickly, but it > would be nice to know if the problem disappears with 3.2. Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and vobject, I'll let you know. :-) > There /has/ been momentum in fixing it. In py3k. Yes, I specifically meant in the 2.x branch. I'm guessing I'll have to stay on 2.x for at least 5 more years, due to the other package dependencies. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On 5/17/2010 2:59 PM, Bill Janssen wrote: Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and vobject, I'll let you know. :-) There /has/ been momentum in fixing it. In py3k. Yes, I specifically meant in the 2.x branch. I'm guessing I'll have to stay on 2.x for at least 5 more years, due to the other package dependencies. I suspect it will be sooner than that, especially if users like you ask/beg/plead with the maintainers of libraries like those you listed to make them work with 3.2. Give your particular reason, that Python3 will work increasingly well with multicore machines. I an sure a couple of such people have posted that they see no reason to upgrade until users start requesting them to. Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
> Not fixing this is a big problem. It relegates the only Python which > will be usable by many many people for many years, 2.x, to a toy > language status on modern machines. It will have threads, but the use > of them, either directly or indirectly by modules you may import, may > cause unpredictable performance problems. People may disagree with this characterization, but if we take that for granted, then, yes, we are willing to accept that as the state of things for the coming years. People running into these problems will have a number of choices to them: switch operating systems (i.e. drop OSX for something that actually works), switch programming languages (i.e. drop Python for something that actually works), switch application architectures (i.e. drop threads for something that actually works), switch to 3.x, or just accept the problem, and hope that the system will find something else to do while switching Python threads. > I'd hate to let a fundamental flaw like this go through simply because > someone somewhere somewhen set a completely synthetic deadline. No, it's not like that. We set the deadline so that we are able to cancel discussions like this one. It would be possible to change the schedule, if we would agree that it was for a good cause - which we don't. > If threading performance wasn't broken on multicore, I'd agree with you. > But right now, *everyone* has to be an expert just to use Python 2.x > effectively on modern multicore hardware Not at all. Just use the multiprocessing module instead, and be done. It's really easy to use if you already managed to understand threads. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Martin v. Löwis wrote: > > I'd hate to let a fundamental flaw like this go through simply because > > someone somewhere somewhen set a completely synthetic deadline. > > No, it's not like that. We set the deadline so that we are able to > cancel discussions like this one. It would be possible to change the > schedule, if we would agree that it was for a good cause - which we don't. I do appreciate that, and also what you and Antoine are saying. > > If threading performance wasn't broken on multicore, I'd agree with you. > > But right now, *everyone* has to be an expert just to use Python 2.x > > effectively on modern multicore hardware > > Not at all. Just use the multiprocessing module instead, and be done. > It's really easy to use if you already managed to understand threads. But that's just the problem. Most folks don't use "threads", they use a higher-level abstraction like the nltk library. Does it use threads? Has its owner ported it to py3k? Has its owner ported it to the multiprocessing module? I have to be an expert to know. I'll stop talking about this now... At least, here. Apparently we only need to fix this for OS X. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Martin v. Löwis wrote: > People running into these problems will have a number of choices to > them: switch operating systems (i.e. drop OSX for something that > actually works), switch programming languages (i.e. drop Python for > something that actually works), switch application architectures (i.e. > drop threads for something that actually works), switch to 3.x, or > just accept the problem, and hope that the system will find something > else to do while switching Python threads. There's even another option: if the new-GIL backport patch in http://bugs.python.org/issue7753 works for you, apply it and run with it (and advocate for a Python 2.8 release to make it more widely available, possibly even contributing the fallback behaviour discussed in the issue to make that situation more likely). Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Terry Reedy wrote: > On 5/17/2010 2:59 PM, Bill Janssen wrote: > > Yes, it would. As soon as I have working 3.x versions of BeautifulSoup, > > PIL, ReportLab, JCC, pylucene, pyglet, nltk, email, epydoc, feedparser, > > dictclient, docutils, hachoir, mutagen, medusa, python-dateutil, and > > vobject, I'll let you know. :-) > > > >> There /has/ been momentum in fixing it. In py3k. > > > > Yes, I specifically meant in the 2.x branch. I'm guessing I'll have to > > stay on 2.x for at least 5 more years, due to the other package > > dependencies. > > I suspect it will be sooner than that, especially if users like you > ask/beg/plead with the maintainers of libraries like those you listed > to make them work with 3.2. Oh, that's the way I like to spend my day (and, as you can tell from this conversation, I'm really good at it :-). Though I will of course do that. But some of these, like JCC+pylucene, nltk, and vobject, were developed with idiosyncratic funding resources which no longer exist. Others, like pyglet, were developed for a completely different purpose, and I doubt the developers care what I want. So, realistically, I doubt it will be less than five years. Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: >> So the patch to the threading code would presumably, for those OSs where >> the capability exists, try to put all created threads in the same >> affinity set. > > This is not really a good idea. There's some code which releases the > GIL, precisely so that you can run several threads (computations) at > once. Somewhat irrelevant given the rest of this thread, but you could potentially deal with that by playing CPU affinity games in the BEGIN/END_ALLOW_THREADS macros. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Florent Xicluna wrote: > I would like to have the possibility to "./configure > --without-broken-GIL" or "./configure --with-bfs-scheduler" in Python > 2.7 like we "./configure --with-computed-gotos" in 3.2. > > It will let the opportunity for the experienced user (or the > distribution packager) to enable the threading optimizations on its > platform, while it preserves the current behaviour by default. > > It will give more chance that people test this experimental configure > option and report any issue they find, so it can be fixed and improved > in the next bugfix version (2.7.1). > Since the 2.7 branch will have long term support, it makes sense to > support multi-core platforms. > > And more users of the fixed GIL means more bugfixes (even for 3.x branch). Would you suggest extending this as far as providing "new GIL" binaries for Windows and Mac OS X? (If that was the case, it begins to sound a lot like http://bugs.python.org/issue7753 with reversion to the old GIL on non-NT/POSIX platforms) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia --- ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Antoine Pitrou wrote: > On Sun, 16 May 2010 15:13:44 PDT > Bill Janssen wrote: > > > > So the patch to the threading code would presumably, for those OSs where > > the capability exists, try to put all created threads in the same > > affinity set. > > This is not really a good idea. There's some code which releases the > GIL, precisely so that you can run several threads (computations) at > once. Could the macro that releases the GIL also release the thread affinity? And the macro that acquires it again set the affinity tag back again? Bill ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
Le lundi 17 mai 2010 à 14:36 -0700, Bill Janssen a écrit : > > Could the macro that releases the GIL also release the thread affinity? We release the GIL in a variety of situations which don't necessarily involve heavy computations (such as: waiting for IO or sleeping). Furthermore, having several threads taking / dropping the GIL would make the affinity setting / unsetting pattern quite chaotic. Really, I think the processor affinity solution can only be application-specific. There doesn't seem to be an easy, generic way of guessing whether some kind of processor affinity should be requested or not. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
> But some of these, like JCC+pylucene, nltk, and vobject, were > developed with idiosyncratic funding resources which no longer exist. > Others, like pyglet, were developed for a completely different > purpose, and I doubt the developers care what I want. So, > realistically, I doubt it will be less than five years. Make it a GSoC project next summer, and you have them ported next fall. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
I would like to restart this thread with 2 notes from the lively discussion: a) Issue 7946 (and this discussion?) concerns Python 3.2 b) The GIL problems are not specific to OSX. The old and new GIL misbehave on GNU/Linux and Windows too. [Putting on anti-frying-pan helmet] Nir ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Fixing the GIL (with a BFS scheduler)
On Mon, May 17, 2010 at 15:05, Antoine Pitrou wrote: > On Mon, 17 May 2010 14:47:25 +0200 > Lennart Regebro wrote: >> On Mon, May 17, 2010 at 14:12, Antoine Pitrou wrote: >> > On Mon, 17 May 2010 08:28:08 +0200 >> > Lennart Regebro wrote: >> >> But the scheduler is so simplistic it ends up fighting with the >> >> OS scheduler, and a large amount of CPU time is used up switching >> >> instead of executing. >> > >> > This is already fixed with py3k. >> >> Are you referring to the "New GIL"? > > Yes. At has been shown, it also in certain cases will race with the OS scheduler, so this is not already fixed, although apparently improved, if I understand correctly. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64 ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
