Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-14 Thread Michael Felt
@Ned - thx for the hints. The bot behavior is "back to normal". Still have to find out why the test_multiprocession_* fail on the bot, but not when I run them manually (on my AIX6.1 build server). As I have time, I'll continue to research. Also, I'll look to a patch to check BOTH include files AN

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-13 Thread Michael Felt
On 11/13/2018 11:07 PM, Michael Felt wrote: > As my system that does not any libexpat does not need this "help", I am > going to guess that since configure finds the include files it builds > pyexpat differently than when the include file is not available. > > I'll research further. (I'll start w

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-13 Thread Michael Felt
On 11/12/2018 3:16 AM, Ned Deily wrote: > On Nov 11, 2018, at 15:27, Michael wrote: >> Anyway, my bot went from having 9 to 10 fails (and 10k lines of output) >> to 20 fails and roughly 22k lines of test. >> >> The last "near normal FAIL" is: >> https://buildbot.python.org/all/#/builders/161/bui

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-11 Thread Ned Deily
On Nov 11, 2018, at 15:27, Michael wrote: > Anyway, my bot went from having 9 to 10 fails (and 10k lines of output) > to 20 fails and roughly 22k lines of test. > > The last "near normal FAIL" is: > https://buildbot.python.org/all/#/builders/161/builds/325/steps/4/logs/stdio > and the first mega

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-11 Thread Michael
On 10/11/2018 23:34, Michael Felt (aixtools) wrote: OK. I spent a bit of time looking at the results, and to my dismay - there was a major shift in the number of errors that the AIX bot returns. I spent about a month of real-time to find corrections to all the open AIX tests - so that the build bo

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-10 Thread Michael Felt (aixtools)
Sent from my iPhone > On 10 Nov 2018, at 21:29, Zachary Ware wrote: > >> On Sat, Nov 10, 2018 at 12:48 PM David Bolen wrote: >> If you look back on the worker there's probably some other build on the >> same worker still technically running that's blocking the rest. It might >> be old enoug

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-10 Thread Zachary Ware
On Sat, Nov 10, 2018 at 12:48 PM David Bolen wrote: > If you look back on the worker there's probably some other build on the > same worker still technically running that's blocking the rest. It might > be old enough to not show on the main summary page, and you won't see if it > just looking at

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-10 Thread David Bolen
If you look back on the worker there's probably some other build on the same worker still technically running that's blocking the rest. It might be old enough to not show on the main summary page, and you won't see if it just looking at a single builder history, so look at the worker in general.

[Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-10 Thread Michael
Just noticed my build-bot (#161) and the other AIX build-bot (#10) are stuck acquiring locks. #161 stuck 5 days (actually, 159, 160, 161, 162 and 163) #10 stuck 2 days (9, 10 and 131) - with 9 pinging builder for 9 hours (builders 4 and 104 are quiet) FYI. p.s. Not unique to AIX - this one is a