Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-11 Thread Michael
On 10/11/2018 23:34, Michael Felt (aixtools) wrote:
OK. I spent a bit of time looking at the results, and to my dismay -
there was a major shift in the number of errors that the AIX bot returns.

I spent about a month of real-time to find corrections to all the open
AIX tests - so that the build bot could be useful in spotting changes
that seriously affected AIX. Without that effort there is not really any
point in running a bot.

Anyway, my bot went from having 9 to 10 fails (and 10k lines of output)
to 20 fails and roughly 22k lines of test.

The last "near normal FAIL" is:
https://buildbot.python.org/all/#/builders/161/builds/325/steps/4/logs/stdio
and the first mega fail is
https://buildbot.python.org/all/#/builders/161/builds/326/steps/4/logs/stdio

No way I am going to research all of those if noone is going to take
review and reject or merge my submissions.

I suppose I could look daily - but it just says fail, so I watch the PRs
instead. The bot just says fail.

Does anyone besides myself care?

Sincerely,
Michael
>
> Sent from my iPhone
>
>> On 10 Nov 2018, at 21:29, Zachary Ware  wrote:
>>
>>> On Sat, Nov 10, 2018 at 12:48 PM David Bolen  wrote:
>>> If you look back on the worker there's probably some other build on the
>>> same worker still technically running that's blocking the rest.  It might
>>> be old enough to not show on the main summary page, and you won't see if it
>>> just looking at a single builder history, so look at the worker in general.
>> David is exactly right; there were hung builds on a few builders.  I
>> thought I'd added configuration to fix that, but apparently it's not
>> working as expected.  I have manually gone through and cancelled the
>> hung builds, so hopefully those builders will catch up soon.
> Thx. My 5 day job finished. Scary though. 20 tests failed, rather than the 
> expected 7. 
>> -- 
>> Zach
>> ___
>> Python-Buildbots mailing list
>> Python-Buildbots@python.org
>> https://mail.python.org/mailman/listinfo/python-buildbots
> ___
> Python-Buildbots mailing list
> Python-Buildbots@python.org
> https://mail.python.org/mailman/listinfo/python-buildbots





signature.asc
Description: OpenPGP digital signature
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots


Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

2018-11-11 Thread Ned Deily
On Nov 11, 2018, at 15:27, Michael  wrote:
> Anyway, my bot went from having 9 to 10 fails (and 10k lines of output)
> to 20 fails and roughly 22k lines of test.
> 
> The last "near normal FAIL" is:
> https://buildbot.python.org/all/#/builders/161/builds/325/steps/4/logs/stdio
> and the first mega fail is
> https://buildbot.python.org/all/#/builders/161/builds/326/steps/4/logs/stdio

Michael,

Thanks for tracking down where the additional failures began.  I took a quick 
look at the buildbot listings for builds 325 and 325 and it seems like all of 
the additional failures that appear in 326 vs 325 are a result of a new build 
failure for the pyexpat module.  That's visible at the end of the "compile" 
stage output:

Failed to build these modules:
pyexpat

and near the top of the test output when the Makefile tries to rebuild any 
failed modules.

Looking at the output of the git stage of 326, the git branch HEAD of the build 
is 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d.  Running git show to get the 
commit message:

$ git show 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d
commit 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d
Author: Gregory P. Smith 
Date:   Wed Oct 17 18:10:46 2018 -0700

bpo-35011: Restore use of pyexpatns.h in libexpat (GH-9939)

Restores the use of pyexpatns.h to isolate our embedded copy of the expat C
library so that its symbols do not conflict at link or dynamic loading time
with an embedding application or other extension modules with their own
version of libexpat.


https://github.com/python/cpython/commit/5dc3f23b5fb0b510926012cb3732dae63cddea60#diff-3afaf7274c90ce1b7405f75ad825f545
 inadvertently removed it when upgrading expat.

That looks rather suspicious!

So I guess the next step would be to try to figure out why that change (or some 
other in the range between buildbot run 325 and 326) causes the pyexpact module 
to stop building.  There are a bunch of ld: 0711-317 ERROR: Undefined symbol 
errors in the link step of the "building 'pyexpat' extension" step.  In the 
"configure" step, it is determined that it is not using an operating system 
provided version of expat:

 checking for --with-system-expat... no

Which is to be expected.  So it should be trying to pick up a version of expat 
that is provided elsewhere outside of the build.  Perhaps there is an issue 
with it.  Or the above change doesn't take something into account.  To proceed 
further, it would be most helpful to have someone with AIX experience and/or 
familiarity with that buildbot's setup.


--
  Ned Deily
  n...@python.org -- []

___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots