Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 04:45, schrieb Brian Quinlan: > On Dec 29, 2010, at 2:55 PM, Victor Stinner wrote: > >> Le mercredi 29 décembre 2010 à 21:49 +0100, "Martin v. Löwis" a écrit : >>> Of course, one may wonder why test_first_completed manages >>> to create 41 SemLock objects, when all it tries to do is two future >>> calls. >> >> More numbers (on Linux): >> >> - Queue: 3 SemLock >> - Condition: 4 SemLock >> - Event: 5 SemLock >> - Call (test_concurrent_futures): 10 SemLock (2 Event) >> - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) >> >> FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, >> you can only create *one* Call object, whereas some tests use 4 Call >> objects or more. > > Great detective work! This would suggest that ProcessPoolExecutors are > useable on FreeBSD 7.2 so long as the user doesn't create more than two > at once (which probably isn't a big deal for most apps). IIUC, this is a system-wide limit. So if you run any non-trivial app a number of times will quickly cause failure. This is just not practical. > So skipping the test is probably the way to go. I'm still -1 on that proposal. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Wiadomość napisana przez Martin v. Löwis w dniu 2010-12-30, o godz. 10:16: > Am 30.12.2010 04:45, schrieb Brian Quinlan: > >> So skipping the test is probably the way to go. > > I'm still -1 on that proposal. I agree with Martin, explanation follows. In general, I'm trying to think as the user: someone wrote a program and said it requires Python 3.2. I expect it to work on Python 3.2 available on my platform. Doesn't matter if it's bundled with the OS or custom built. When a module is not available, this should fail loudly and as fast as possible. This is currently the case with conditionally built modules like curses, sqlite3, tkinter and others. From the user's perspective: the import fails. From the administrator's perspective: the build reports skipped modules. The admin can then alter the environment to make the build behave as expected, and then run the tests to confirm it works. From an OS maintainer perspective it's even better: a module can be made available optionally with explicitly set dependencies and required configuration. In our specific case a bunch of things should be determined first: 1. Does it still fail on FreeBSD 7.3+? 2. Why is the semaphore limit so low in the first place? 3. What is a reasonable number for that limit? That should include situations where people run multiple applications. 4. What other configuration should be done to ensure the module won't break on runtime. Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. One problem is that the administrator can make the OS compatible/incompatible on runtime (with sysctl). ISTM concurrent.futures was included a bit too early in the distribution. The module has problems not only on BSD [1]. Fortunately, there is visible action from Brian, Philip and David to make the problems go away. I'm sure it will be fixed pretty soon. Ultimately I also agree with Martin that an implementation using SysV IPC would probably be better from the user's perspective. As he said, even if the underlying code is not as pretty as with the POSIX APIs, once it's written and tested nobody cares because it's not exposed. However, it's far too late to change that now for 3.2. [1] On RedHat/CentOS the test freezes badly (issue 10517). -- Best regards, Łukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Thu, 30 Dec 2010 16:04:16 +0100 Łukasz Langa wrote: > > Some answers Philip gave already. Knowing all these things would let us > decide whether switching the module off on systems that don't meet the > requirements is okay and can we get away with just documenting how to make it > work. I really don't think it is our job to maintain a list of OS/versions which work and don't work. At best, multiprocessing when imported could try to create a single semaphore, and raise an ImportError if such creating fails (which it does, IIUC, under some old FreeBSDs without a specific kernel tweak). > ISTM concurrent.futures was included a bit too early in the distribution. > The module has problems not only on BSD [1]. I'm not sure concurrent.futures is the culprit, rather than multiprocessing itself (or perhaps even some core Python functionality). Actually, the threading-based part of concurrent.futures probably works fine. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 16:40, schrieb Antoine Pitrou: > On Thu, 30 Dec 2010 16:04:16 +0100 > Łukasz Langa wrote: >> >> Some answers Philip gave already. Knowing all these things would let us >> decide whether switching the module off on systems that don't meet the >> requirements is okay and can we get away with just documenting how to make >> it work. > > I really don't think it is our job to maintain a list of OS/versions > which work and don't work. Of course not. I would propose a dynamic test: check how many POSIX semaphores the installation supports, and fail if it's less than 200 (say). >> ISTM concurrent.futures was included a bit too early in the distribution. >> The module has problems not only on BSD [1]. > > I'm not sure concurrent.futures is the culprit, rather than > multiprocessing itself (or perhaps even some core Python functionality). > Actually, the threading-based part of concurrent.futures probably works > fine. Well, "the culprit" really is FreeBSD. However, concurrent.futures apparently makes freehanded use of semaphores, in a way that the FreeBSD authors didn't expect them to be used (if they expected them to be used at all, that is). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
> > I'm not sure concurrent.futures is the culprit, rather than > > multiprocessing itself (or perhaps even some core Python functionality). > > Actually, the threading-based part of concurrent.futures probably works > > fine. > > Well, "the culprit" really is FreeBSD. However, concurrent.futures > apparently makes freehanded use of semaphores, in a way that the > FreeBSD authors didn't expect them to be used (if they expected them > to be used at all, that is). It seems to be multiprocessing as much as concurrent.futures itself. Apparently a basic multiprocessing queue already uses two (interprocess) locks and one semaphore. And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
> And, again, the threading-based API in concurrent.futures should work > fine anyway. Do you suggest we selectively disable the process-based > API? Yes. Importing concurrent.futures.process should fail. Unfortunately, it's imported from __init__.py, so either we change the API to move the executors to the submodules, or we let creation of process pools fail (rather than the import). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
R. David Murray writes: > I thought Amaury was saying it was harder than svnmerge, not harder > than patching by hand (ie: that it *was* patching by hand, including > .rej files and the lack of tracking). I also heard Georg say that this > should be fixable, and that he's partly fixed the tracking issue, I don't see why the tracking issue is any different than it would be for svn. If you're actually merging, either a dummy merge (what git calls an "ours merge") or reverting unwanted patches will "block" them, and hg will keep track of the ones that have been merged. If you're actually cherry-picking, then you have to keep a separate database of the patches that have been picked and those that are blocked, but this has been done successfully for years with svnmerge, right? > but not the patch/merge issue (and that doing so will require a change > in the hg core). I don't think so. For merges, there isn't a problem. For cherypicking, I haven't thought carefully about this, but ISTM that "hg export | hg import; merge post-patch /dev/null source" should give the traditional conflict markers. This will require a bit of care to get the files to merge right, since there will in general be multiple files that fail to patch, but the names can be fished out of the .rej file(s). Getting the source files will also be mildly tricky, since they live in a different branch, and aren't available in your local repository. It will also require a bit of care to get the commit log etc right. But I don't think it's that hard, conceptually. Of course it's preferable to get this feature in hg itself, but I don't think we need to wait for hg/maintain a fork. > Well, considering that the transition is "soon", the fact that it > is a SMOP is a concern :) Sure, but in this crowd, I wouldn't waste a good worry on this particular SMOP. > Well, there will be *some* workflow change since we're talking about > committing to 3.2 maint before the new trunk instead of vice versa. > But you are right that that is, mostly, a detail. AFAIK that's not hg-driven, though. > I'm not as convinced that changes in workflow will be that minimal, > though. I don't have much experience with the dvcses yet, but what > experience I have had leads me to think that understanding the DAG is > a non-trivial and necessary mental leap and does affect the workflow. Yes, no, and yes. That is, although understanding it is nontrivial, and once you do it will affect your workflow, it is unnecessary. The Emacs crew have proved that to my satisfaction; there are a bunch of folks there who understand DAGs quite well, but there are also a bunch who just don't want to know -- and they're doing fine. True, making the most of dVCS requires "understanding the DAG." Adapting an existing complex workflow to a new dVCS's commands also requires understanding the DAG. But using the adapted workflow simply requires learning new names for old operations. Annoying, but it will make a fair number of core devs quite happy. > Well, I hope you are right. I'm looking forward to the new version of > the test repository and doing some real world experiments. Yup. It's always risky to predict, especially the future, but I'm pretty confident that things will work out. We do need to (1) make sure we do everything for hg that we've always done for svn, (2) work out some features that hg doesn't have yet (Windows EOLs), and (3) listen carefully to those who are testing out the new repository -- there are always surprises in this kind of thing. But Python does that kind of thing very well. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
> 1. Does it still fail on FreeBSD 7.3+? Yes, it still fails. The limits (30 semaphores) haven't changed. It also remains untunable. > 2. Why is the semaphore limit so low in the first place? I don't know - (Free)BSD is in the tradition of disliking SysV inventions, and POSIX inventions unless they originate from FreeBSD. This is polemics, of course :-) In any case, SysV IPC doesn't seem much better. In SysV, you allocate semaphores in sets. In FreeBSD 8.1, in the standard setting, you can have up to 10 semaphore identifiers, with up to 60 semaphores each. That may sound reasonable, except that there is also a limit on the total number of semaphores of 60. So it seems that switching to SysV IPC wouldn't actually improve things (except that these are tunable parameters already, so changing them requires only a reboot, not a recompile of the kernel). > 3. What is a reasonable number for that limit? That should include > situations where people run multiple applications. POSIX specifies that the minimum acceptable value for the SEM_NSEMS_MAX setting is _POSIX_SEM_NSEMS_MAX, which in turn must be at least 256. We could argue that we refuse to use POSIX semaphores if the system doesn't conform to POSIX, i.e. has semaphore limit of less than 256. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 30.12.2010 18:54, schrieb Stephen J. Turnbull: > R. David Murray writes: > > > I thought Amaury was saying it was harder than svnmerge, not harder > > than patching by hand (ie: that it *was* patching by hand, including > > .rej files and the lack of tracking). I also heard Georg say that this > > should be fixable, and that he's partly fixed the tracking issue, > > I don't see why the tracking issue is any different than it would be > for svn. If you're actually merging, either a dummy merge (what git > calls an "ours merge") or reverting unwanted patches will "block" > them, and hg will keep track of the ones that have been merged. Are you still talking about merges from 3.x to 2.7? Then I don't think what you say is actually true. hg will *not* be able to track the ones that get merged, and will *not* be able to block by means of dummy merges. Perhaps we aren't "actually" merging, I suppose. > If > you're actually cherry-picking, then you have to keep a separate > database of the patches that have been picked and those that are > blocked, but this has been done successfully for years with svnmerge, > right? [Ok. So "cherry-picking" is not a special case of "merging", or "actually merging", I presume] Wrong. For subversion, merge tracking is built into the tools that we use for merging (i.e. svnmerge). For hg (IIUC), the standard procedure that people use for merging (or "cherry-picking") apparently does not track what has been merged (or "cherry-picked"). > > Well, considering that the transition is "soon", the fact that it > > is a SMOP is a concern :) > > Sure, but in this crowd, I wouldn't waste a good worry on this > particular SMOP. Given how little enthusiasm this crowd has in actually helping the migration, I doubt they show any enthusiasm in writing such tools. > > Well, there will be *some* workflow change since we're talking about > > committing to 3.2 maint before the new trunk instead of vice versa. > > But you are right that that is, mostly, a detail. > > AFAIK that's not hg-driven, though. It is. So far, we have *always* merged from the development branch to the maintenance branch (i.e. "backported"), although there has been opposition from a number of committers to this workflow in recent years. I personally consider this more appropriate: with backporting, you can defer decision to backport until the original change has been confirmed as correct. With constant forward-porting, you risk breaking your maintenance branch with incorrect changes. > > > I'm not as convinced that changes in workflow will be that minimal, > > though. I don't have much experience with the dvcses yet, but what > > experience I have had leads me to think that understanding the DAG is > > a non-trivial and necessary mental leap and does affect the workflow. > > But using the adapted workflow simply requires learning new names for > old operations. Annoying, but it will make a fair number of core devs > quite happy. I think the new workflow will simply result in (even) less care for the maintenance branches than currently. Some people just refuse to patch multiple branches, and will continue to do so. While it was previously possible to backport, it won't be that easy anymore in the future, so it just won't be done. This actually reduces workload, but also reduces quality. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On Fri, 31 Dec 2010 02:54:26 +0900, "Stephen J. Turnbull" wrote: > I don't see why the tracking issue is any different than it would be > for svn. If you're actually merging, either a dummy merge (what git Martin already said most of what I would have in response to your post. > For cherypicking, I haven't thought carefully about this, but ISTM > that "hg export | hg import; merge post-patch /dev/null source" should > give the traditional conflict markers. This will require a bit of > care to get the files to merge right, since there will in general be > multiple files that fail to patch, but the names can be fished out of > the .rej file(s). Getting the source files will also be mildly > tricky, since they live in a different branch, and aren't available in > your local repository. It will also require a bit of care to get the > commit log etc right. But I don't think it's that hard, conceptually. > > Of course it's preferable to get this feature in hg itself, but I > don't think we need to wait for hg/maintain a fork. The fact that I really haven't a clue what you are talking about here means that I for one am not likely to be helping develop that tool, at least not any time soon. So I hope there are people who understand this stuff who will make it work for the rest of us. > > Well, considering that the transition is "soon", the fact that it > > is a SMOP is a concern :) > > Sure, but in this crowd, I wouldn't waste a good worry on this > particular SMOP. Talent is one thing, available time, as you pointed out about yourself, is a different matter. I'm confident we can make this all work. The only question is when. -- R. David Murray www.bitdance.com ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Hi, One thing confuses me in this thread: Do the problems come from merging across different versions (i.e. different syntax and module names), or are they regular problems that come up because people don’t want to use a merge tool? I for one immensely like Mercurial’s merge and utterly dislike Subversion’s (certainly because I’ve learned a good merge tool and don’t have a good editor to handle files with conflict markers). > There is surely a better way to work with maintenance branches! > PEP374 suggests to first modify the oldest release, but this does not > seems right to me (I have three reasons in mind) Can you give them? Regards ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On Thu, 30 Dec 2010 20:57:11 +0100 Éric Araujo wrote: > Hi, > > One thing confuses me in this thread: Do the problems come from merging > across different versions (i.e. different syntax and module names), or > are they regular problems that come up because people don’t want to use > a merge tool? I for one immensely like Mercurial’s merge and utterly > dislike Subversion’s (certainly because I’ve learned a good merge tool > and don’t have a good editor to handle files with conflict markers). IIUC this is not about merges, this is about transplants (or cherry-picking). It would be unannoying to lose svnmerge's relative convenience in that matter. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/30 Antoine Pitrou : > On Thu, 30 Dec 2010 20:57:11 +0100 > Éric Araujo wrote: >> Hi, >> >> One thing confuses me in this thread: Do the problems come from merging >> across different versions (i.e. different syntax and module names), or >> are they regular problems that come up because people don’t want to use >> a merge tool? I for one immensely like Mercurial’s merge and utterly >> dislike Subversion’s (certainly because I’ve learned a good merge tool >> and don’t have a good editor to handle files with conflict markers). > > IIUC this is not about merges, this is about transplants (or > cherry-picking). It would be unannoying to lose svnmerge's relative > convenience in that matter. I assume you meant "annoying"? -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Le jeudi 30 décembre 2010 à 14:24 -0600, Benjamin Peterson a écrit : > 2010/12/30 Antoine Pitrou : > > On Thu, 30 Dec 2010 20:57:11 +0100 > > Éric Araujo wrote: > >> Hi, > >> > >> One thing confuses me in this thread: Do the problems come from merging > >> across different versions (i.e. different syntax and module names), or > >> are they regular problems that come up because people don’t want to use > >> a merge tool? I for one immensely like Mercurial’s merge and utterly > >> dislike Subversion’s (certainly because I’ve learned a good merge tool > >> and don’t have a good editor to handle files with conflict markers). > > > > IIUC this is not about merges, this is about transplants (or > > cherry-picking). It would be unannoying to lose svnmerge's relative > > convenience in that matter. > > I assume you meant "annoying"? Well, yes. I should stop twisting my negatives (whatever it means). Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Martin v. Löwis wrote: And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Yes. Importing concurrent.futures.process should fail. If I understand correctly, it is possible to adjust BSD so that this will work -- as a user I would much rather be informed of that, even as just a caution, than to have Python not have the functionality even after I had fixed the OS problems. Mind you, I have no idea how hard it would be to make this happen in the module in an intelligent way. ~Ethan~ ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
On Thu, 30 Dec 2010 12:36:47 -0800 Ethan Furman wrote: > Martin v. Löwis wrote: > >> And, again, the threading-based API in concurrent.futures should work > >> fine anyway. Do you suggest we selectively disable the process-based > >> API? > > > > Yes. Importing concurrent.futures.process should fail. > > If I understand correctly, it is possible to adjust BSD so that this > will work -- as a user I would much rather be informed of that, even as > just a caution, than to have Python not have the functionality even > after I had fixed the OS problems. > > Mind you, I have no idea how hard it would be to make this happen in the > module in an intelligent way. As I said, we can just emit a Warning instead of raising an exception. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 30.12.2010 19:31, schrieb "Martin v. Löwis": >> But using the adapted workflow simply requires learning new names for >> old operations. Annoying, but it will make a fair number of core devs >> quite happy. > > I think the new workflow will simply result in (even) less care for the > maintenance branches than currently. Some people just refuse to > patch multiple branches, and will continue to do so. While it was > previously possible to backport, it won't be that easy anymore in > the future, so it just won't be done. I refuse to believe that we cannot make our committers (in particular the active ones, which is quite a small percentage) follow such simple rules, especially when they have to learn a new VCS anyway... Also, it's not really necessary for everyone to merge every change from maintenance to trunk: a) they can do multiple commits on the same subject and merge once, and b) if the change is small and no problems can be expected from merging, merging can also be done by others, just like the mass merging we had once from trunk -> py3k (just more convenient for the one doing the merge). Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 19:17, schrieb "Martin v. Löwis": >> 1. Does it still fail on FreeBSD 7.3+? > > Yes, it still fails. The limits (30 semaphores) haven't > changed. It also remains untunable. > >> 2. Why is the semaphore limit so low in the first place? > > I don't know - (Free)BSD is in the tradition of disliking > SysV inventions, and POSIX inventions unless they originate from > FreeBSD. This is polemics, of course :-) BTW - can anyone contribute data points from other *BSDs? Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
> Also, it's not really necessary for everyone to merge every change from > maintenance to trunk: a) they can do multiple commits on the same > subject and merge once, and b) if the change is small and no problems can > be expected from merging, merging can also be done by others, just like > the mass merging we had once from trunk -> py3k (just more convenient for > the one doing the merge). But you can't use Mercurial's merge functionality for that, right? You have to use some kind of transplant/cherry-picking to merge from the 3.3 branch to the 3.2 branch, right? Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 30.12.2010 20:57, schrieb Éric Araujo: > Hi, > > One thing confuses me in this thread: Do the problems come from merging > across different versions (i.e. different syntax and module names), or > are they regular problems that come up because people don’t want to use > a merge tool? Neither nor. They come from "hg merge" being useless when it comes to having code that is meant to live both on 2.7, 3.2 (perhaps) and 3.3. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
Am 30.12.2010 21:36, schrieb Ethan Furman: > Martin v. Löwis wrote: >>> And, again, the threading-based API in concurrent.futures should work >>> fine anyway. Do you suggest we selectively disable the process-based >>> API? >> >> Yes. Importing concurrent.futures.process should fail. > > If I understand correctly, it is possible to adjust BSD so that this > will work -- as a user I would much rather be informed of that, even as > just a caution, than to have Python not have the functionality even > after I had fixed the OS problems. That doesn't exclude each other. The module will fail on default (misconfigured) systems, and work on correctly-configured systems. See my patch for details. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
Am 30.12.2010 22:38, schrieb "Martin v. Löwis": >> Also, it's not really necessary for everyone to merge every change from >> maintenance to trunk: a) they can do multiple commits on the same >> subject and merge once, and b) if the change is small and no problems can >> be expected from merging, merging can also be done by others, just like >> the mass merging we had once from trunk -> py3k (just more convenient for >> the one doing the merge). > > But you can't use Mercurial's merge functionality for that, right? > You have to use some kind of transplant/cherry-picking to merge from the > 3.3 branch to the 3.2 branch, right? Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction you need cherry-picking, yes. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
> BTW - can anyone contribute data points from other *BSDs? I don't have an installation of OpenBSD, but... In FreeBSD, POSIX semaphores are implemented in sys/kern/uipc_sem.c. In http://www.openbsd.org/cgi-bin/cvsweb/src/sys/kern/ that file doesn't exist. Also, in FreeBSD's limits.h, _POSIX_SEM_NSEMS_MAX is defined (surprisingly to 256); in http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/include/limits.h?rev=1.15;content-type=text/plain this constant doesn't appear. So ISTM that OpenBSD doesn't implement POSIX semaphores. IIUC, this means that the multiprocessing module won't be fully functional, and its tests (and the concurrent.futures tests) will be skipped. NetBSD apparently supports sem_open since 2.0: http://netbsd.gw.com/cgi-bin/man-cgi?sem_open++NetBSD-current According to http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/uipc_sem.c?rev=1.22&content-type=text/x-cvsweb-markup&only_with_tag=MAIN SEM_MAX is 128 since 2007, and dynamically adjustable (no reboot). Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
On 12/30/2010 4:44 PM, Georg Brandl wrote: But you can't use Mercurial's merge functionality for that, right? You have to use some kind of transplant/cherry-picking to merge from the 3.3 branch to the 3.2 branch, right? Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction you need cherry-picking, yes. I have the impression that Benjamin plans to continue 3.1 bug maintenance for months after 3.2 final rather than issue the last bug fix the traditional 1 week after. That would make the sequence 3.1->3.2->3.3 or 3.1<-3.2<-3.3 or ... . The transition would be a lot easier, I think, if 3.1.4 was released along with 3.2, so most of us would never have to bother with 3.1 and hg together. -- Terry Jan Reedy ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] Demo is gone
According to the consensus (and loosely following the Google spreadsheet I created for that purpose), I have removed the Demo directory from py3k. Most demos have been deleted; some have been moved into Tools or into the docs as an example. If I removed something you think should have stayed, please speak up. I kept a few less-useful ones in a new directory called Tools/demo that aims to give a broad overview of what you can do with a few lines of Python. Please feel free to remove and/or add demos there, if you find better ones. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Backport troubles with mercurial
2010/12/30 Terry Reedy : > On 12/30/2010 4:44 PM, Georg Brandl wrote: > >>> But you can't use Mercurial's merge functionality for that, right? >>> You have to use some kind of transplant/cherry-picking to merge from the >>> 3.3 branch to the 3.2 branch, right? >> >> Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction >> you need cherry-picking, yes. > > I have the impression that Benjamin plans to continue 3.1 bug maintenance > for months after 3.2 final rather than issue the last bug fix the > traditional 1 week after. That would make the sequence 3.1->3.2->3.3 or > 3.1<-3.2<-3.3 or ... . The transition would be a lot easier, I think, if > 3.1.4 was released along with 3.2, so most of us would never have to bother > with 3.1 and hg together. I will try to release 3.1.4 soon after 3.2 is released, but February and the spring in general will be quite busy for me. -- Regards, Benjamin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD
"Martin v. Löwis" writes: >> 1. Does it still fail on FreeBSD 7.3+? > > Yes, it still fails. The limits (30 semaphores) haven't > changed. It also remains untunable. Yeah, my recollection about 7.3 appears to have been remembering when the kernel module was included by default as opposed to needing to be explicitly loaded. From the older discussion (http://mail.python.org/pipermail/python-dev/2010-November/105380.html) it would appear that 7.x remains fixed sans a kernel build (not necessarily a big deal for FreeBSD users), but 8.1+ can be tuned with sysctl. -- David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
2010/12/28 Lukas Lueg > Consider the following code: > > def foobar(x): >for i in range(5): >x[i] = i > > The bytecode in python 2.7 is the following: > > 2 0 SETUP_LOOP 30 (to 33) > 3 LOAD_GLOBAL 0 (range) > 6 LOAD_CONST 1 (5) > 9 CALL_FUNCTION1 > 12 GET_ITER >>> 13 FOR_ITER16 (to 32) > 16 STORE_FAST 1 (i) > > 3 19 LOAD_FAST1 (i) > 22 LOAD_FAST0 (x) > 25 LOAD_FAST1 (i) > 28 STORE_SUBSCR > 29 JUMP_ABSOLUTE 13 >>> 32 POP_BLOCK >>> 33 LOAD_CONST 0 (None) > 36 RETURN_VALUE > > Can't we optimize the LOAD_FAST in lines 19 and 25 to a single load > and put the reference twice on the stack? There is no way that the > reference of i might change in between the two lines. Also, the > load_fast in lne 22 to reference x could be taken out of the loop as x > will always point to the same object Yes, you can, but you need: - a better AST evaluator (to mark symbols/variables with proper attributes); - a better optimizer (usually located on compile.c) which has a "global vision" (not limited to single instructions and/or single expressions). It's not that simple, and the results aren't guaranteed to be good. Also, consider that Python, as a dynamic-and-not-statically-compiled language need to find a good trade-off between compilation time and execution. Just to be clear, a C program is usually compiled once, then executed, so you can spend even *hours* to better optimize the final binary code. With a dynamic language, usually the code is compiled and the executed as needed, in "realtime". So it isn't practical neither desirable having to wait too much time before execution begins (the "startup" problem). Python stays in a "gray area", because modules are usually compiled once (when they are first used), and executed many times, but it isn't the only case. You cannot assume that optimization techniques used on other (static) languages can be used/ported in Python. Cesare ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Possible optimization for LOAD_FAST ?
2010/12/29 "Martin v. Löwis" > Am 28.12.2010 18:08, schrieb Lukas Lueg: > > Also, the > > load_fast in lne 22 to reference x could be taken out of the loop as x > > will always point to the same object > > That's not true; a debugger may change the value of x. > > Regards, > Martin OK, but is it mandatory? For example, in the above code, I can unroll the loop because I found that range is the usual built-in, 5 is a low-enough constant, and the body is made by a simple statement. Another example. I can totally remove the variable i, just using the stack, so a debugger (or, in general, having the tracing enabled) cannot even find something to change about it. And so on with other optimization examples that can be possible. Are they "legal" with Python? I think that we need to make it clear what happens in such cases. My idea is that it should be made implementation-specific. What happens with local variables and the generated code must depend on the specific compiler & virtual machine, in order to have a greater flexibility. IMHO the most important thing should be that, under normal conditions, the executed code have the expected behavior. Cesare ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
