Re: [Python-Dev] Stable buildbots
On Sun, Nov 14, 2010 at 12:40 PM, David Bolen wrote: > Antoine Pitrou writes: > >> (even though the Windows buildbots give >> a rather unconventional meaning to the word "stability"). > > Nag, nag, nag :-) > > There's been a bit of an uptick in the past few weeks with hung > python_d processes (not a new issue, but it ebbs and flows), so I'm > going to try to pull together a monitor script this weekend to start > killing them off automatically. Should at least get rid of some of > the low hanging fruit that interferes with subsequent builds. Do we have any idea why the workaround to avoid the popup windows stopped working? (assuming it ever worked reliably - I thought it did, but that impression may have been incorrect) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
On Sun, Nov 14, 2010 at 1:10 PM, Terry Reedy wrote: > Much better except possible for \n after 'summary:' That extra line break helps more for multi-line checkin messages (which happen reasonably often). Doesn't really bother me either way - I'm mainly looking for info on who has the ability to change the format in the first place :) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
Nick Coghlan writes: > Do we have any idea why the workaround to avoid the popup windows > stopped working? (assuming it ever worked reliably - I thought it did, > but that impression may have been incorrect) Oh, the pop-up handling for the RTL dialogs still seems to be working fine (at least I haven't seen any since I put it in place). That, plus the original buildbot tweaks to block any OS popups still looks solid for avoiding any dialogs that block a test process. This is a completely separate issue, though probably around just as long, and like the popup problem its frequency changes over time. By "hung" here I'm referring to cases where something must go wrong with a test and/or its cleanup such that a python_d process remains running, usually several of them at the same time. So I end up with a bunch of python_d processes in the background (but not with any dialogs pending), which eventually cause errors during attempts the next time the same builder is used since the file remains in use. I expect some of this may be the lack of a good process group cleanup under Windows, though the root cause may not be unique to Windows. I see something very similar reasonable frequency on my OSX Tiger buildbot as well. But since the filesystem there can let the build tree get cleaned and rebuilt even with a stranded executable, the impact is minimal on subsequent tests than on Windows, though the OSX processes do burn a ton of CPU. I run a script on OSX to kill them off, but that was quick to whip up since in those cases the stranded processes all end up getting owned by init so it's a simple ps grep and kill. In the Windows case I'll probably just set a time limit so if the processes have been around more than a few hours I figure they're safe to kill. -- David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of Win32 ANSI API
> If the code is currently working and isn't a security hole, then we > obviously don't "have to". > Apparently several developers "want to", which is different. In case the motivation for that isn't clear: it would produce a significant code reduction, and therefore ease maintenance. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of Win32 ANSI API
> We should also keep in mind that *Microsoft* have chosen to keep the > bytes Win32 APIs around, despite their flaws, all in the name of > backwards compatibility. Of course, Microsoft is in a different position. If they remove a functionality in some release, their users typically can't go back and continue to use the old version - at least not on the same computer. For Python, it's different: our users can go back to use an old version if the new one breaks their applications. And we do break applications from time to time, most notably with the introduction of Python 3. > While the goal of nudging third party > developers towards the superior Unicode APIs is an admirable one, it > is still the case that there is a *lot* of ASCII-only code out there. The question is: is there also a lot of ASCII-only Python 3 software out there? And would developers of such software have difficulties to port it to a Unicode file name API. > E.g. applications could easily be storing filenames in an ASCII only > datastore that provides them back to the application as bytes in 3.x. That's speculation. My speculation would be that authors of such a datastore find that they can't even print the data anymore in a reasonable way, so they changed their API to return strings (i.e. decoding from ASCII) when they ported it to Python 3. They wouldn't even consider it a change, because it returned strings all the time, and now Python 3 has a different string type. >> If you drop code without first deprecating it, chances are it will >> hurt someone. That's why having a deprecation period is the rule we >> usually follow (most of the time :-)). I'm in favor of deprecating it first. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
> This is a completely separate issue, though probably around just as > long, and like the popup problem its frequency changes over time. By > "hung" here I'm referring to cases where something must go wrong with > a test and/or its cleanup such that a python_d process remains > running, usually several of them at the same time. So I end up with a > bunch of python_d processes in the background (but not with any > dialogs pending), which eventually cause errors during attempts the > next time the same builder is used since the file remains in use. This is what kill_python.exe is supposed to solve. So I recommend to investigate why it fails to kill the hanging Pythons. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
Am 14.11.2010 09:25, schrieb Nick Coghlan: > On Sun, Nov 14, 2010 at 1:10 PM, Terry Reedy wrote: >> Much better except possible for \n after 'summary:' > > That extra line break helps more for multi-line checkin messages > (which happen reasonably often). Doesn't really bother me either way - > I'm mainly looking for info on who has the ability to change the > format in the first place :) See http://hg.python.org/hooks/ You should have push permissions to that repository. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
"Martin v. Löwis" writes: > This is what kill_python.exe is supposed to solve. So I recommend to > investigate why it fails to kill the hanging Pythons. Yeah, I know, and I can't say I disagree in principle - not sure why Windows doesn't let the kill in that module work (or if there's an issue actually running it under all conditions). At the moment though, I do know that using the sysinternals pskill utility externally (which is what I currently do interactively) definitely works so to be honest, automating that is a guaranteed bang for buck at this point with no analysis involved. Looking into kill_python or its use can be a follow-on. -- David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unexpected traceback/stack behavior with chained exceptions (issue 1553375)
On Sun, Nov 14, 2010 at 1:40 PM, R. David Murray wrote: > Issue 1553375 [1] proposes a patch to add an 'allframes' option to the > traceback printing and formatting routines so that the full traceback > from the top of the execution stack down to the exception is printed, > instead of just from the point where the exception is caught down to > the exception. This is useful when the reason you are capturing the > traceback is to log it, and you have several different points in your > application where you do such traceback logging. You often really want > to know the entire stack in that case; logging only from the capture > point down can lose important debugging information depending on how > the application is structured. > > The patch seems to work well, except for one problem that I don't have > enough CPython internals knowledge to understand. If the traceback we > are printing has a chained traceback, the resulting full traceback shows > the line that is printing the traceback instead of the line from the 'try' > block. (It prints the expected line if there is no chained traceback). > > So, is this a failure in my understanding of how tracebacks are supposed > to work, or a bug in how chained tracebacks are constructed? It looks to me like you're grabbing a reference to a frame that is currently executing and that frame has moved on since the exception was thrown (to your exception handler). The print_stack() call in the patch then accurately reflects this. The other thing to keep in mind is that the exception currently being handled is the *last* one produced by _iter_chain - all of the rest have already been caught and handled, it was the handlers for those that raised the subsequent exceptions in the chain. Basically, the whole patch strikes me as fundamentally misguided. If someone wants this information in their exception handler, they should put a print_stack() with the appropriate header information after the print_exception() call rather than trying to embed it in the display of the exception information. logging could also gain an independent "stack_trace=True" option to request inclusion of a stack trace independently of whether or not exception information is included. (Side note: there's a typo in the format_tb docstring claiming it is a wrapper around extract_stack - that's incorrect, it is a wrapper around extract_tb) Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Removal of Win32 ANSI API
On Sun, Nov 14, 2010 at 8:14 PM, "Martin v. Löwis" wrote: > I'm in favor of deprecating it first. Aye. I've made the best case I could for keeping it, and even I don't find it terribly convincing. So deprecation for 3.2 sound like a reasonable option. Regards, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
On Sun, Nov 14, 2010 at 8:20 PM, "Martin v. Löwis" wrote: > Am 14.11.2010 09:25, schrieb Nick Coghlan: >> On Sun, Nov 14, 2010 at 1:10 PM, Terry Reedy wrote: >>> Much better except possible for \n after 'summary:' >> >> That extra line break helps more for multi-line checkin messages >> (which happen reasonably often). Doesn't really bother me either way - >> I'm mainly looking for info on who has the ability to change the >> format in the first place :) > > See > > http://hg.python.org/hooks/ > > You should have push permissions to that repository. Thanks - it will give me a chance to use Hg for something meaningful as well. Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
On Sun, Nov 14, 2010 at 8:20 PM, "Martin v. Löwis" wrote: > See > > http://hg.python.org/hooks/ > > You should have push permissions to that repository. I suspect my hg-fu is inadequate to at this point - I get an 'access to repository "hg.python.org/hooks" not permitted' error when I try to push the modified file to "ssh://[email protected]/hooks". (I actually got the same error when cloning, but if I understand hg correctly, it shouldn't matter that my clone came from the http URL rather than the ssh one). My username and email address in my hgrc file match those in Dirkjan's author map, so I'm not sure what's going on there. The change I tried to make was to replace the last couple of lines of the header creation mail.py's incoming() function with the following 3 lines: body += log.splitlines()[:-2] body += ['summary:\n ' + ctx.description(), ''] body += ['files:\n ' + '\n '.join(ctx.files()), ''] Cheers, Nick. -- Nick Coghlan | [email protected] | Brisbane, Australia ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
Am 14.11.2010 13:39, schrieb Nick Coghlan: > On Sun, Nov 14, 2010 at 8:20 PM, "Martin v. Löwis" wrote: >> See >> >> http://hg.python.org/hooks/ >> >> You should have push permissions to that repository. > > I suspect my hg-fu is inadequate to at this point - I get an 'access > to repository "hg.python.org/hooks" not permitted' error when I try to > push the modified file to "ssh://[email protected]/hooks". Martin told you only half the truth: the SSH URL is (currently) . I think we will change that to remove the /repos/ part before going live with the cpython repo, but the hg username remains, corresponding to the pythondev username for SVN. > (I actually > got the same error when cloning, but if I understand hg correctly, it > shouldn't matter that my clone came from the http URL rather than the > ssh one). That's correct. > My username and email address in my hgrc file match those in Dirkjan's > author map, so I'm not sure what's going on there. The usernames and email addresses you use for commits don't matter; as long as you can connect via SSH you can push commits with any author. cheers, Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Issues 9931 and 9055 - test_ttk_guionly and buildbot run as a service
On 12 November 2010 17:07, Terry Reedy wrote: > On 11/12/2010 3:44 AM, Paul Moore wrote: >> >> Hi, >> My buildbot has been failing for some time because of these 2 issues, >> both related to the fact that tests are hanging when run as a service >> (and hence have no display to open GUI elements on). Both issues have >> patches, and as far as I am aware, the patches fix the issues >> reasonably well. What can I do to move these 2 issues forwards? As >> things stand, my buildbot is not providing a lot of value on the 3.x >> branch :-( > > http://bugs.python.org/issue9055 > is marked as a 2.7 issue only, perhaps fixed by Tim Golden's committed > patches. Should it be re-versioned for 3.1/2? There is no patch file > attached, though perhaps the code in Yamamoto's message is meant as such > (but for which version?). So the first thing you could do is clarify the > current status and remaining issue on the tracker. Ah, sorry. I misremembered the history - you are right, I suspect this is fixed (at least to the extent that my buildbot isn't permanently red :-)) On rereading, I get the impression that a cleaner fix may be possible by using the ideas in the patch for 9931, but that's probably for another time. > http://bugs.python.org/issue9931 > by Yamamoto is marked for all 3 versions. It seems to be a similar issue, > though marked 'test' rather than 'ctypes'. It does have a patch by him > apparently based on his previous comments. The issue has no responses and > needs a patch review. So the first thing you could do is to provide one;-). > If it looks great (no changes that you can think of) and works great, say > so. Then it can move on to commit review stage. OK, thanks. I'll see if I can provide a review, and see how it goes from there. Really, it's not that urgent that this gets fixed in the wider scheme of things - but as my buildbot is a bit useless while the problem remains, I'm motivated to do what I can to work on it. I'm just a little limited in what I can do, hence the request for suggestions. > PS. Providing links like the above makes it easier for multiple people to > take a look and respond. You're right, and I apologise for that. I sent the email in a hurry and didn't consider others before sending. Paul ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
On 14 November 2010 02:40, David Bolen wrote: > There's been a bit of an uptick in the past few weeks with hung > python_d processes (not a new issue, but it ebbs and flows), so I'm > going to try to pull together a monitor script this weekend to start > killing them off automatically. Should at least get rid of some of > the low hanging fruit that interferes with subsequent builds. My buildslave (x86 XP-5, see http://www.python.org/dev/buildbot/buildslaves/moore-windows) runs buildbot as a service. I set it up that way as I assumed that would be the most sensible approach to avoid manual intervention on reboots, keeping a user session permanently running, etc. But it seems that there are a few areas where things don't work quite right when run from a service (see, for example, http://bugs.python.org/issue9931) and I assumed that some of my hung python_d processes were related to that. Do you run your slave as a service? (And for that matter, what do other Windows slave owners do?) Are there any "best practices" for ongoing admin of a Windows buildslave that might be worth collecting together? (I'll try to put some notes on what I've found together - maybe a page on the Python wiki would be the best place to collect them). Paul. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
> I suspect my hg-fu is inadequate to at this point - I get an 'access > to repository "hg.python.org/hooks" not permitted' error when I try to > push the modified file to "ssh://[email protected]/hooks". Try ssh://[email protected]/repos/hooks I think this is something that needs to be fixed: I fail to see the point of having this extra repos/ directory in the path (even though it's certainly useful to have them all in a separate directory on disk). It's also unfortunate that hg complains it can't give access to /hooks, when the problem really is that the repository doesn't exist. I guess this is because it tries to create it, and then finds that it can't. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
On Sun, 14 Nov 2010 19:27:22 +0100 "Martin v. Löwis" wrote: > > I suspect my hg-fu is inadequate to at this point - I get an 'access > > to repository "hg.python.org/hooks" not permitted' error when I try to > > push the modified file to "ssh://[email protected]/hooks". > > Try > > ssh://[email protected]/repos/hooks > > I think this is something that needs to be fixed: I fail to see the > point of having this extra repos/ directory in the path (even though > it's certainly useful to have them all in a separate directory on disk). IIUC, "repos/hooks" is interpreted as a relative path to the "hg" user's HOME. The "ssh://" scheme executes remote hg over an ssh session, I don't think there's any additional magic. Regards Antoine. ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
>> I think this is something that needs to be fixed: I fail to see the >> point of having this extra repos/ directory in the path (even though >> it's certainly useful to have them all in a separate directory on disk). > > IIUC, "repos/hooks" is interpreted as a relative path to the "hg" > user's HOME. The "ssh://" scheme executes remote hg over an ssh > session, I don't think there's any additional magic. Correct. However, this just means that additional magic is required. Regards, Martin ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] unexpected traceback/stack behavior with chained exceptions (issue 1553375)
Nick Coghlan gmail.com> writes: > of the exception information. logging could also gain an independent > "stack_trace=True" option to request inclusion of a stack trace > independently of whether or not exception information is included. Good point, Nick. There are times when you'd want to know how you got to a certain point in code, irrespective of whether any exception occurred. So your suggestion makes sense, and I'll try and see if I can get it into 3.2. Another benefit of this is that a user only gets this if they want it; if I were to use the allframes flag in logging, then everyone would get the print_stack() even if they didn't want it. Regards, Vinay Sajip ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] PEP 385: Formatting of Hg checkin notifications
Am 14.11.2010 19:35, schrieb Antoine Pitrou: > On Sun, 14 Nov 2010 19:27:22 +0100 > "Martin v. Löwis" wrote: >> > I suspect my hg-fu is inadequate to at this point - I get an 'access >> > to repository "hg.python.org/hooks" not permitted' error when I try to >> > push the modified file to "ssh://[email protected]/hooks". >> >> Try >> >> ssh://[email protected]/repos/hooks >> >> I think this is something that needs to be fixed: I fail to see the >> point of having this extra repos/ directory in the path (even though >> it's certainly useful to have them all in a separate directory on disk). > > IIUC, "repos/hooks" is interpreted as a relative path to the "hg" > user's HOME. The "ssh://" scheme executes remote hg over an ssh > session, I don't think there's any additional magic. There is; we already have a custom authorized_keys command in place to call the hg-ssh wrapper, and all that's needed is to customize that command a bit more. Georg ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] Stable buildbots
Paul Moore writes: > Do you run your slave as a service? (And for that matter, what do > other Windows slave owners do?) Are there any "best practices" for > ongoing admin of a Windows buildslave that might be worth collecting > together? (I'll try to put some notes on what I've found together - > maybe a page on the Python wiki would be the best place to collect > them). I've always run my slave interactively under Windows (well, started it interactively). Not sure if I tried a service in the beginning or not, it was a while ago. So your slave is probably the guinea pig for service operation. There is http://wiki.python.org/moin/BuildbotOnWindows (for which I can't take any credit). It could probably use a little love and updating, and it's largely aimed at setting things up, but not as much operating it. I think the only stuff I'm doing on my slave above and beyond the basic setup is a small patch to buildbot (circa 2007, couldn't get it back upstream at the time) to use SetErrorMode to disable OS pop-ups, and the AutoIt script (from earlier this year) to auto-acknowledge C RTL pop-ups. The kill script in this thread as a safety net above kill_python would be a third tweak. There was a buildbot fix for uploading that was only needed for the short-lived MSI generation, and which I think later buildbot versions have their own changes for. I'd be happy to work with you if you're willing to combine/edit our bits of information. Probably something we can take off-list, so just let me know. -- David ___ Python-Dev mailing list [email protected] http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
