Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks
On 10/11/2018 23:34, Michael Felt (aixtools) wrote: OK. I spent a bit of time looking at the results, and to my dismay - there was a major shift in the number of errors that the AIX bot returns. I spent about a month of real-time to find corrections to all the open AIX tests - so that the build bot could be useful in spotting changes that seriously affected AIX. Without that effort there is not really any point in running a bot. Anyway, my bot went from having 9 to 10 fails (and 10k lines of output) to 20 fails and roughly 22k lines of test. The last "near normal FAIL" is: https://buildbot.python.org/all/#/builders/161/builds/325/steps/4/logs/stdio and the first mega fail is https://buildbot.python.org/all/#/builders/161/builds/326/steps/4/logs/stdio No way I am going to research all of those if noone is going to take review and reject or merge my submissions. I suppose I could look daily - but it just says fail, so I watch the PRs instead. The bot just says fail. Does anyone besides myself care? Sincerely, Michael > > Sent from my iPhone > >> On 10 Nov 2018, at 21:29, Zachary Ware wrote: >> >>> On Sat, Nov 10, 2018 at 12:48 PM David Bolen wrote: >>> If you look back on the worker there's probably some other build on the >>> same worker still technically running that's blocking the rest. It might >>> be old enough to not show on the main summary page, and you won't see if it >>> just looking at a single builder history, so look at the worker in general. >> David is exactly right; there were hung builds on a few builders. I >> thought I'd added configuration to fix that, but apparently it's not >> working as expected. I have manually gone through and cancelled the >> hung builds, so hopefully those builders will catch up soon. > Thx. My 5 day job finished. Scary though. 20 tests failed, rather than the > expected 7. >> -- >> Zach >> ___ >> Python-Buildbots mailing list >> Python-Buildbots@python.org >> https://mail.python.org/mailman/listinfo/python-buildbots > ___ > Python-Buildbots mailing list > Python-Buildbots@python.org > https://mail.python.org/mailman/listinfo/python-buildbots signature.asc Description: OpenPGP digital signature ___ Python-Buildbots mailing list Python-Buildbots@python.org https://mail.python.org/mailman/listinfo/python-buildbots
Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks
On Nov 11, 2018, at 15:27, Michael wrote: > Anyway, my bot went from having 9 to 10 fails (and 10k lines of output) > to 20 fails and roughly 22k lines of test. > > The last "near normal FAIL" is: > https://buildbot.python.org/all/#/builders/161/builds/325/steps/4/logs/stdio > and the first mega fail is > https://buildbot.python.org/all/#/builders/161/builds/326/steps/4/logs/stdio Michael, Thanks for tracking down where the additional failures began. I took a quick look at the buildbot listings for builds 325 and 325 and it seems like all of the additional failures that appear in 326 vs 325 are a result of a new build failure for the pyexpat module. That's visible at the end of the "compile" stage output: Failed to build these modules: pyexpat and near the top of the test output when the Makefile tries to rebuild any failed modules. Looking at the output of the git stage of 326, the git branch HEAD of the build is 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d. Running git show to get the commit message: $ git show 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d commit 9d4712bc8f26bf1d7e626b53ab092fe030bcd68d Author: Gregory P. Smith Date: Wed Oct 17 18:10:46 2018 -0700 bpo-35011: Restore use of pyexpatns.h in libexpat (GH-9939) Restores the use of pyexpatns.h to isolate our embedded copy of the expat C library so that its symbols do not conflict at link or dynamic loading time with an embedding application or other extension modules with their own version of libexpat. https://github.com/python/cpython/commit/5dc3f23b5fb0b510926012cb3732dae63cddea60#diff-3afaf7274c90ce1b7405f75ad825f545 inadvertently removed it when upgrading expat. That looks rather suspicious! So I guess the next step would be to try to figure out why that change (or some other in the range between buildbot run 325 and 326) causes the pyexpact module to stop building. There are a bunch of ld: 0711-317 ERROR: Undefined symbol errors in the link step of the "building 'pyexpat' extension" step. In the "configure" step, it is determined that it is not using an operating system provided version of expat: checking for --with-system-expat... no Which is to be expected. So it should be trying to pick up a version of expat that is provided elsewhere outside of the build. Perhaps there is an issue with it. Or the above change doesn't take something into account. To proceed further, it would be most helpful to have someone with AIX experience and/or familiarity with that buildbot's setup. -- Ned Deily n...@python.org -- [] ___ Python-Buildbots mailing list Python-Buildbots@python.org https://mail.python.org/mailman/listinfo/python-buildbots