Re: [Python-buildbots] Stable builders

[David Bolen]

Just an FYI related to these:

> "Iffy" (have failures that look related to the slave, but are
> platforms that should be stable.  If I reshuffle right now, they'll be
> marked unstable):
> bolen-ubuntu
> bolen-windows
> bolen-windows8
> bolen-windows10

I've watched them consistently failing for a while now, but whenever I
peek at selected failures it doesn't seem to be slaved related, as
opposed to true test failures (albeit those I may not understand a
root cause behind).  So I've assumed they were issues in-development.
Certainly bolen-ubuntu has traditionally been extremely robust in
terms of passing tests (unlike windows sometimes) so its failures are
usually actual failures.  Of course, in that case I suppose a new test
might have added host requirements for certificate validation that
requires something to change on the build slave.

For windows, I sort of assumed the getaddrinfo failures might have
been tests that just didn't work properly on newer windows versions,
but have to admit I did not try to dig deeper.  I don't think there's
anything I can do at the slave level to impact that.

If the stable list just means passing under the current code, the
above set definitely should be excluded.  I'm just not sure they
aren't representing actual issues for the most part.

-- David
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Stable builders

[David Bolen]

Zachary Ware
 writes:

> I'm not sure about bolen-ubuntu, but I figured its issues are most
> likely slave- (or slave-environment-) related due to the fact that all
> branches are failing and the other Ubuntu bots are fine.  I did notice
> that bolen-ubuntu seems to have a pretty old OpenSSL (1.0.1f), might
> upgrading it be enough?  I don't think we support any but the most
> recent release of any given version of OpenSSL.

Hmm, at the system level it's 1.0.1f, but that's up to date for the
platform (Ubuntu 14.04.3) so I'd think should work.

I'll do a general update and see if that cleans anything up.

>> For windows, I sort of assumed the getaddrinfo failures might have
>> been tests that just didn't work properly on newer windows versions,
>> but have to admit I did not try to dig deeper.  I don't think there's
>> anything I can do at the slave level to impact that.
>
> I'm suspicious of Azure for those.  My 8.1 bot has no problem with the
> test that's causing problems on both Azure-based bots.

Hmm, good point about your bot - I should have thought to look at that.
Ok, I'll dig in further.  My 8.1 bot had been passing previously (after
resolving some of the new build stuff) so figured something new had been
introduced, but it's a good point that it might be a collision between
the tests and some Azure network restrictions.

-- David

___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Windows 2.7 Builders

[David Bolen]

Interesting - I probably would have assumed removing 2010 was safe for
2.7 as well.  However, all of my 2.7 builders (XP/7/8/10) currently
have both 2008 and 2010 so any dependency on 2010 would go unnoticed.

Assuming the 2010 dependency is unintentional, or we specifically want
to test without it, the XP slave no longer builds anything but 2.7, so
I could actually uninstall 2010 from that machine without affecting
any other builders, or requiring any special configuration to avoid a
2010 installation.

-- David

On Thu, Nov 3, 2016 at 2:40 PM, Jeremy Kloth  wrote:
>
> Hello,
>
> I was recently cleaning up my buildbot, clearing space, removing
> unused software, ..., and decided to remove MSVC 2010 as no supported
> Python uses that compiler.  Well, to my surprise, actually I forgot,
> you do need MSVC 2010 *AND* 2008 to build 2.7 by default.  It has been
> since reinstalled and the buildbot is chugging along happily.
>
> With that, do we have any buildbots that use *just* 2008?  Or does the
> 2.7 compiler package from Microsoft contain the needed bits that are
> provided by installing Visual Studio 2010?
>
> I think my buildslave has enough resources to support another builder
> for that configuration (VC9 only) if that is something we are
> interested in doing.
>
> Thanks
>
> --
> Jeremy Kloth
> ___
> Python-Buildbots mailing list
> Python-Buildbots@python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Windows 2.7 Builders

[David Bolen]

Zachary,

> I'd like to just add a builder to your XP bot that builds in PC/VS9.0
> in addition to the regular builder in PCbuild.  That will allow us to
> keep track of whether a change breaks one build or both.  Once I have
> a VS9-only builder set up, if you want to remove VS2010 from any of
> your other bots, let me know and I'll switch its 2.7 builder to use
> VS9-only.

The additional builder sounds fine.  I don't have any particular need
to remove 2010 anywhere, but wasn't sure if perhaps by having it still
installed we were glossing over something on the buildslaves that a
third party might run into, which doesn't seem to be the case.

In terms of switching any other builders over, I suppose the intent
should be to use whatever scripts are used for the final python.org
builds, so we test what will be distributed.  I realize the actual
compiler in use has far more ramifications than the build process, but
might as well stay as aligned as possible.  I'm guessing that's
PCBuild, so no change to the existing builders, and the buildslaves
keep the 2010 dependency for 2.7.

But for third party independent builds, it appears like the 2.7.12
documentation (https://docs.python.org/2/using/windows.html) still
lists PCBuild and 2008.  So perhaps that should be updated to PC/VS9.0
similar to the earlier versions?  At which point, keeping at least one
builder using PC/VS9.0 definitely seems like a good idea.

-- David
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Windows 2.7 Builders

[David Bolen]

BTW, just to follow up on my last message:

> But for third party independent builds, it appears like the 2.7.12
> documentation (https://docs.python.org/2/using/windows.html) still
> lists PCBuild and 2008.

I should have checked first, but the readme.txt in the PCBuild
directory seems to cover this in detail, so at worst, the online
document is a minor issue.  Although interestingly, the online doc
points to PC/readme.txt which doesn't cover any of this.  But I'm sure
someone wanting to compile things is going to figure it out once they
have the source in front of them.

And it's still probably a good idea to have a PC/VS9.0 based builder.

-- David
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Git installation and roll call

[David Bolen]

Git has been installed on all of my buildslaves.  And I don't believe any
lost messages in the past for me.

-- David

On Wed, Dec 7, 2016 at 11:20 PM, Zachary Ware 
wrote:

> Hi all,
>
> It has come to my attention that one of our members has been trying to
> send mail to the list and it has been silently disappearing.  To
> determine whether this is a widespread issue, please reply-all to this
> message, sending to both the list and me.  I'll compare replies I
> receive to the archives.  If you've also sent messages that were never
> replied to, please mention that as well.
>
> We'll also take this opportunity to ensure that all of our buildslaves
> have Git installed in preparation for the GitHub switchover.  Please
> note whether Git is installed on your slave(s) in your reply.
>
> Thanks,
> --
> Zach
> ___
> Python-Buildbots mailing list
> Python-Buildbots@python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Buildbot: buildslave angelico-debian-amd64 was lost

[David Bolen]

On Fri, Feb 10, 2017 at 5:40 PM, Chris Angelico  wrote:
> I didn't see anything fall over on my end. Was this disconnect related
> to the hg->git changeover, and if not, is there something I should
> check?

Probably not, or at least I haven't seen any disconnects across any of
my buildbots.  The slave page does show yours as disconnect however,
so I suspect restarting it would clear that up, regardless of the
original fault.

For what it's worth, I have on rare occasions in the past had one of
my slaves reflected as disconnected at the server but the slave still
believes it is connected.  My working theory has been some sort of
network issue that breaks or blocks traffic and the server just
notices first, but I never really dug too deeply, just opting to
restart the slave.

-- David
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Moving Python.org to Buildbot 9 + Python 3

[David Bolen]

On Fri, Oct 6, 2017 at 4:33 PM, Craig Rodrigues  wrote:

> Based on my experiments, I think it is safe to perform the migration as
> follows:
>
> 1.   Update the buildbot master to buildbot 9 + Python 3.  Leave the
> buildbot workers/slaves alone.  They should be
>   able to successfully connect to the new buildbot master and work.
>
> 2.   For the buildbot worker/slave maintainers, give them the following
> options:
>
>   a.  Upgrade the buildbot worker to buildbot 9 + Python 3(MOST
> PREFERRED)
>   b.  Upgrade the buildbot worker to buildbot 9 + Python 2(LESS
> PREFERRED, BUT WILL WORK)
>   c.  Leave the buildbot slave alone at buildbot 8 + Python 2  (LEAST
> PREFERRED, BUT WILL WORK)

Yes, my own preference would be to do workers last.  It sounds to me
like most of the benefits of this change accrue on the master side
(eliminating our custom version) and at least for my part, I'd prefer
not to be critical path to allowing that upgrade to proceed.

Over a longer period, I'm curious about the preferences - are there
concrete benefits in our environment of 0.9 over 0.8 on the worker
side?  E.g., what does upgrading bring on the worker side?  In some
cases an upgrade is trivial, but I've still got a few local tweaks on the
Windows side that, while probably not too difficult to port, incur cost
only worth doing if there is a concrete requirement or benefit.

Oh, and I can offer any of my Windows workers for interoperability
testing if you'd like me to temporarily configure them to an
additional master.  At least one of them (XP) is probably not worth
the effort of getting to (a) vs. (b).

-- David
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Moving Python.org to Buildbot 9 + Python 3

[David Bolen]

Yes they look to be ok now.  Although for future reference, both Windows XP
and 7 workers did initially break - apparently due to an incompatibility
when using Python < 2.7 on the worker (I have 2.6 on those machines).  The
master is sending remote methods down using unicode for method names.  So
2.7+ appears to be a worker requirement in this mixed-mode setup (though I
doubt any new worker would try to use 2.6).

For now, I've locally modified my version of twisted on each buildbot to
deal with that, pending time to have them upgraded to 2.7 (or possibly in
the Win7 case, just go to 3.x).

I'm not sure if that might be the same issue with kloth-win64.

-- David

On Fri, Oct 13, 2017 at 3:59 PM, Craig Rodrigues 
wrote:

> On Fri, Oct 6, 2017 at 1:46 PM, David Bolen  wrote:
>
>>
>> Oh, and I can offer any of my Windows workers for interoperability
>> testing if you'd like me to temporarily configure them to an
>> additional master.  At least one of them (XP) is probably not worth
>> the effort of getting to (a) vs. (b).
>>
>
>
> I don't think I need this right now.
> If I look at:
> http://buildbot.python.org/all/#/workers
>
> Your windows buildbot workers seem to be working fine with the new
> buildbot master.
>
> This is one buildbot worker kloth-win64 which is not working.
> I don't know what the problem with that one is.
>
> --
> Craig
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Moving Python.org to Buildbot 9 + Python 3

[David Bolen]

Does testing include a mixture with a master under 3.x?  I only ask since
I'd guess that a master under 2.x might be sending parameters down as
strings rather than unicode.

Anyway, no, I hadn't tried a new worker, because both the same and an
earlier buildbot version was working fine on my Win8/10 workers, so I
focused more on other environmental differences, and since time was tight,
once I identified a workaround, I stopped for the time being.

I did misspeak in the last note - the issue was a unicode name for a
keyword parameter remote function call, not the function name itself.  The
actual failure occurs inside twisted while dispatching the remote call from
the master.  It takes a PB-transmitted dictionary for the arguments, using
either apply() or **kwargs against them directly.  That fails with a
"TypeError: keywords must be strings" under 2.6 but 2.7 is fine.

Since the dispatch failure occurs during the dispatch to buildbot-worker,
but before it gets any chance to run, I'm not sure that it could change
things.  Unless the master changes something in how it transmits request
arguments that needed a newer worker on the client to identify.

Oh, I did also peek at the latest twisted source but didn't see any
differences in that path at a quick glance (aside from switching from
apply() to **kw) that would seem to be able to resolve this, but that's
something else I could test.

Or I could just move off of 2.6 which I last updated to in 2009 :-)

-- David

On Fri, Oct 13, 2017 at 4:17 PM, Craig Rodrigues 
wrote:

> Did you try leaving things on Python 2.6, but upgrading the worker to
> buidlbot-worker 0.9.12 ?
> buildbot-worker is still tested on Python 2.6.
>
> --
> Craig
>
> On Fri, Oct 13, 2017 at 1:07 PM, David Bolen  wrote:
>
>> Yes they look to be ok now.  Although for future reference, both Windows
>> XP and 7 workers did initially break - apparently due to an incompatibility
>> when using Python < 2.7 on the worker (I have 2.6 on those machines).  The
>> master is sending remote methods down using unicode for method names.  So
>> 2.7+ appears to be a worker requirement in this mixed-mode setup (though I
>> doubt any new worker would try to use 2.6).
>>
>> For now, I've locally modified my version of twisted on each buildbot to
>> deal with that, pending time to have them upgraded to 2.7 (or possibly in
>> the Win7 case, just go to 3.x).
>>
>> I'm not sure if that might be the same issue with kloth-win64.
>>
>> -- David
>>
>>
>> On Fri, Oct 13, 2017 at 3:59 PM, Craig Rodrigues 
>> wrote:
>>
>>> On Fri, Oct 6, 2017 at 1:46 PM, David Bolen  wrote:
>>>
>>>>
>>>> Oh, and I can offer any of my Windows workers for interoperability
>>>> testing if you'd like me to temporarily configure them to an
>>>> additional master.  At least one of them (XP) is probably not worth
>>>> the effort of getting to (a) vs. (b).
>>>>
>>>
>>>
>>> I don't think I need this right now.
>>> If I look at:
>>> http://buildbot.python.org/all/#/workers
>>>
>>> Your windows buildbot workers seem to be working fine with the new
>>> buildbot master.
>>>
>>> This is one buildbot worker kloth-win64 which is not working.
>>> I don't know what the problem with that one is.
>>>
>>> --
>>> Craig
>>>
>>
>>
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Internal buildbot error?

[David Bolen]

It does seem like it might be a race condition of some sort - or at least
an unhandled case - in Twisted's process handling.

It's in the writer path, so I wonder if the umask operation completes
quickly enough, perhaps the output handle is in fact invalid by the time
the buildbot parent gets to that section of code.  The umask step looks
like it's only used by the DMG builder, not for regular test builds.

It's fairly uncommon - just twice so far across all branches under the new
master.  To be honest, we could probably just eliminate that step at this
point from the DMG builds (not sure it serves much purpose) or maybe add a
small sleep along with the umask to ensure the child process is around long
enough for the parent to complete the setup.

 -- David

On Thu, Dec 21, 2017 at 5:27 PM, Victor Stinner 
wrote:

> Hi,
>
> What is the following failure? Can it be an internal buildbot bot?
>
> http://buildbot.python.org/all/#/builders/69/builds/75
>
> error in RunProcess._startCommand
> Traceback (most recent call last):
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/buildslave/runprocess.py",
> line 426, in start
> self._startCommand()
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/buildslave/runprocess.py",
> line 550, in _startCommand
> usePTY=self.usePTY)
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/buildslave/runprocess.py",
> line 572, in _spawnProcess
> processProtocol, uid, gid, childFDs)
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/twisted/internet/process.py",
> line 747, in __init__
> parentFD, forceReadHack=True)
> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/
> python2.7/site-packages/twisted/internet/process.py",
> line 156, in __init__
> if not stat.S_ISFIFO(os.fstat(self.fileno()).st_mode):
> OSError: [Errno 9] Bad file descriptor
>
> Traceback (most recent call last):
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 500, in errback
> self._startRunCallbacks(fail)
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 567, in _startRunCallbacks
> self._runCallbacks()
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 653, in _runCallbacks
> current.result = callback(current.result, *args, **kw)
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 1442, in gotResult
> _inlineCallbacks(r, g, deferred)
> ---  ---
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> twisted/internet/defer.py",
> line 1386, in _inlineCallbacks
> result = g.send(result)
>   File "/data/buildbot/venv/lib/python3.4/site-packages/
> buildbot/process/buildstep.py",
> line 705, in run
> results = yield self._start_deferred
> builtins.TypeError: Can't convert 'bytes' object to str implicitly
>
> Victor
> ___
> Python-Buildbots mailing list
> Python-Buildbots@python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Internal buildbot error?

[David Bolen]

BTW, I've found some old emails that the umask step was added as part of
some debugging back in 2010 when the DMG builder was first getting set up,
but the original issue has long since been resolved, so it doesn't really
serve any purpose and might as well get removed.

-- David

On Thu, Dec 21, 2017 at 9:26 PM, David Bolen  wrote:

> It does seem like it might be a race condition of some sort - or at least
> an unhandled case - in Twisted's process handling.
>
> It's in the writer path, so I wonder if the umask operation completes
> quickly enough, perhaps the output handle is in fact invalid by the time
> the buildbot parent gets to that section of code.  The umask step looks
> like it's only used by the DMG builder, not for regular test builds.
>
> It's fairly uncommon - just twice so far across all branches under the new
> master.  To be honest, we could probably just eliminate that step at this
> point from the DMG builds (not sure it serves much purpose) or maybe add a
> small sleep along with the umask to ensure the child process is around long
> enough for the parent to complete the setup.
>
>  -- David
>
> On Thu, Dec 21, 2017 at 5:27 PM, Victor Stinner 
> wrote:
>
>> Hi,
>>
>> What is the following failure? Can it be an internal buildbot bot?
>>
>> http://buildbot.python.org/all/#/builders/69/builds/75
>>
>> error in RunProcess._startCommand
>> Traceback (most recent call last):
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/buildslave/runprocess.py",
>> line 426, in start
>> self._startCommand()
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/buildslave/runprocess.py",
>> line 550, in _startCommand
>> usePTY=self.usePTY)
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/buildslave/runprocess.py",
>> line 572, in _spawnProcess
>> processProtocol, uid, gid, childFDs)
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/twisted/internet/process.py",
>> line 747, in __init__
>> parentFD, forceReadHack=True)
>> File "/Library/Frameworks/Python.framework/Versions/2.7/lib/pytho
>> n2.7/site-packages/twisted/internet/process.py",
>> line 156, in __init__
>> if not stat.S_ISFIFO(os.fstat(self.fileno()).st_mode):
>> OSError: [Errno 9] Bad file descriptor
>>
>> Traceback (most recent call last):
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 500, in errback
>> self._startRunCallbacks(fail)
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 567, in _startRunCallbacks
>> self._runCallbacks()
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 653, in _runCallbacks
>> current.result = callback(current.result, *args, **kw)
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 1442, in gotResult
>> _inlineCallbacks(r, g, deferred)
>> ---  ---
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/twisted/
>> internet/defer.py",
>> line 1386, in _inlineCallbacks
>> result = g.send(result)
>>   File "/data/buildbot/venv/lib/python3.4/site-packages/buildbot/
>> process/buildstep.py",
>> line 705, in run
>> results = yield self._start_deferred
>> builtins.TypeError: Can't convert 'bytes' object to str implicitly
>>
>> Victor
>> ___
>> Python-Buildbots mailing list
>> Python-Buildbots@python.org
>> https://mail.python.org/mailman/listinfo/python-buildbots
>>
>
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

[Python-buildbots] bolen-ubuntu worker updated to 18.04

[David Bolen]

While on my near term to-do list, Ned's note helped give me a bit of a
nudge, so...

The bolen-ubuntu worker has been updated to Ubuntu 18.04 LTS.  This
restores the SSL test coverage, plus includes the libraries to pick up the
previously skipped uuid and ctypes modules.  At this point, of the possible
modules, it should just be tkinter that isn't covered.

This update also switches the worker from 32-bit to 64-bit - mostly due to
it being simpler, but probably also more useful at this point.

-- David
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] bolen-ubuntu worker updated to 18.04

[David Bolen]

Thanks - I hadn't even thought about the prefix.

I had an experimental Xvfb setup when first configuring the buildbot, but
something still broke during the tests.  Unfortunately at this point I
can't remember quite what it was and/or if it was platform/dist specific.
But yes, time permitting I'd like to try again, and will keep your offer in
mind.

-- David

On Wed, May 30, 2018 at 11:56 PM Zachary Ware 
wrote:

> On Wed, May 30, 2018 at 10:30 PM, Berker Peksağ 
> wrote:
> > On Thu, May 31, 2018 at 4:52 AM, Ned Deily  wrote:
> >> Excellent news, David!  Thank you so for much for your generous
> buildbot support over many years!  It is much appreciated.
> >
> > +1, thank you David!
>
> Agreed!
>
> Also, I've now updated the builder name to "AMD64" rather than "x86".
>
> By the way, I've had success with running Xvfb on my Gentoo bot for
> tkinter testing.  If you're interested in setting that up, I'm happy
> to share any relevant details to help :)
>
> --
> Zach
> ___
> Python-Buildbots mailing list
> Python-Buildbots@python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] [Python-Buildbots] - some/many bots stuck acquiring locks

[David Bolen]

If you look back on the worker there's probably some other build on the
same worker still technically running that's blocking the rest.  It might
be old enough to not show on the main summary page, and you won't see if it
just looking at a single builder history, so look at the worker in general.

For example, it looks like there's a 3.6 build from 5 days ago, which
appears stuck in test_multiprocessing_fork for the past 123 hours.  So the
later builds are just waiting for the worker to be available.

I've seen the rare multiprocessing test hang on some of my slower Windows
workers.  I think only maybe within the past 5-6 months and probably less
than a few times overall.  I don't know why the regular test timeouts fail
to interrupt things.  But if you manually kill the stuck test process on
the worker, everything should free up for the rest.  Not sure if a master
side cancellation works or not.

The other cases are appear stuck in builds but in different places. The ARM
worker has a 3.x build  from 10 days ago stuck in test_pydoc.  The other
AIX worker is stuck in a checkout step somehow.  But in all cases, it's not
the waiting builds at issue, but the hung build that blocks the rest.

-- David

On Sat, Nov 10, 2018 at 10:00 AM Michael  wrote:

> Just noticed my build-bot (#161) and the other AIX build-bot (#10) are
> stuck acquiring locks.
>
> #161 stuck 5 days (actually, 159, 160, 161, 162 and 163)
>
> #10 stuck 2 days (9, 10 and 131) - with 9 pinging builder for 9 hours
> (builders 4 and 104 are quiet)
>
> FYI.
>
> p.s. Not unique to AIX - this one is acquiring locks for 9 days!
> https://buildbot.python.org/all/#/builders/8
>
> Michael
>
>
> ___
> Python-Buildbots mailing list
> Python-Buildbots@python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

[Python-buildbots] Windows 8.1 worker (bolen-worker8) retirement

[David Bolen]

My Windows 8.1 worker got hit on Wednesday by some "encrypt the world"
ransomware.  I'm not quite sure the path by which it reached the Azure VM,
but regardless, it's toast.

I don't think I see enough value in re-installation, given the existence of
the Windows 10 worker, plus I can probably bump some Windows 10 worker
resources with the budget freed up by Windows 8.  So I'm planning on
letting the Windows 8 worker end its run at this point.

-- David
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

Re: [Python-buildbots] Current build stuck for four days(!)

[David Bolen]

If you're signed in with your github.com account, then on the buildbot web
interface you should be able to stop the build as builder owner (if your
github.com account matches the worker information and is in the python
group - or something like that).  Although I suppose in this case it may
depend on how its hung.

In most cases for me, I usually just kill the test process locally on the
builder - that'll fail that build abruptly but then everything just moves
on.

-- David

On Sun, Jun 16, 2019 at 9:20 PM Chris Angelico  wrote:

> Turns out that this isn't a massive queue of builds happening, but
> just one single build.
>
> https://buildbot.python.org/all/#/builders/211/builds/46
>
> The builder is saturating one CPU core and has been doing so for
> literally days. I don't think it's going to get anywhere at this
> point. Is there a way to bomb the build and tell it to start over?
>
> ChrisA
> ___
> Python-Buildbots mailing list
> Python-Buildbots@python.org
> https://mail.python.org/mailman/listinfo/python-buildbots
>
___
Python-Buildbots mailing list
Python-Buildbots@python.org
https://mail.python.org/mailman/listinfo/python-buildbots

[Python-buildbots] Re: Two failing buildbots in cpython master

[David Bolen]

This began in response to a change committed for bpo-39401
.  It appears the the intent is to
remove the Windows 7 builders completely from master and subsequent
branches as Python 3.9+ no longer needs to support Windows 7 given its
release will be after the Windows 7 EOL.  The builder removal just hasn't
happened yet.

-- David

On Sat, Feb 8, 2020 at 12:52 AM Kyle Stanley  wrote:

> The following buildbots have been consistently failing across unrelated
> commits in cpython's master branch, during their compilation stages: "AMD64
> Windows7 SP1 3.x" (https://buildbot.python.org/all/#/builders/81) and "x86
> Windows7 3.x" (https://buildbot.python.org/all/#/builders/150). Based on
> their logs, they both seem to have been failing consistently for the last
> 10 days.
>
> Was there some regression introduced between the last successful build (
>
> https://github.com/python/cpython/commit/0cd5bff6b7da3118d0c5a88fc2b80f80eb7c3059
> )
> to the first failing build (
>
> https://github.com/python/cpython/commit/6a65eba44bfd82ccc8bed4b5c6dd6637549955d5
> ),
> or is it unrelated? If it's unrelated, can we temporarily remove those two
> buildbots from the post-commit hooks to cpython's master branch until the
> issue is resolved?
> ___
> Python-Buildbots mailing list -- python-buildbots@python.org
> To unsubscribe send an email to python-buildbots-le...@python.org
> https://mail.python.org/mailman3/lists/python-buildbots.python.org/
>
___
Python-Buildbots mailing list -- python-buildbots@python.org
To unsubscribe send an email to python-buildbots-le...@python.org
https://mail.python.org/mailman3/lists/python-buildbots.python.org/

[Python-buildbots] Re: Buildbots and 2.7

[David Bolen]

In my experience no, workers don't remove build trees automatically, it's
something you have to do manually.

You do get a warning at startup about those that are no longer needed
though.

-- David

On Thu, May 21, 2020 at 3:24 PM David Edelsohn  wrote:

> One of my Z systems recently ran out of disk space.  It appears that
> builders for 3.9 were added, builders for 2.7 were decommissioned, but
> the 2.7 directories were not removed.
>
> Are 2.7 directories suppose to be removed automatically?
>
> Thanks, David
> ___
> Python-Buildbots mailing list -- python-buildbots@python.org
> To unsubscribe send an email to python-buildbots-le...@python.org
> https://mail.python.org/mailman3/lists/python-buildbots.python.org/
> Member address: db3l@gmail.com
>
___
Python-Buildbots mailing list -- python-buildbots@python.org
To unsubscribe send an email to python-buildbots-le...@python.org
https://mail.python.org/mailman3/lists/python-buildbots.python.org/
Member address: arch...@mail-archive.com

[Python-buildbots] Re: buildmaster worker naming

[David Bolen]

With respect to the current Win10 worker, the older system status looks
like my fault - it appears stuck waiting for a manual reboot for prior
updates (I guess at some point it stopped doing it automatically), so I'll
get that caught up.

Personally, I'd prefer not to encode additional release details in the
worker name itself, leaving that stable, but it is a detail that could be
included in the worker info field, so I'll get that adjusted too.

-- David


On Sun, Mar 28, 2021 at 12:29 PM Jeremy Kloth 
wrote:

> I'm looking to add another Windows 10 worker to the "buildbot fleet".
> Since Win10 updates semi-regularly, is it an issue if the worker name
> (BuilderConfig.name) gets changed over the lifetime of the worker?
> Currently it would be "Windows10 20H2", but once the next update is
> released, the name should be updated to "Windows10 21H1".
>
> Mostly I'm looking to differentiate the would be two Win10 workers,
> but there is a bit of practicality to this as well.  Each update (or
> version, as it were) does bring changes and features to the base OS,
> which can be noticed by test_locale failing
> (https://bugs.python.org/issue37945) on recent versions, but not on
> the current Windows 10 worker (at version 1803).
>
> Related to that, if renaming a worker is OK over its lifetime, then
> would it be permissible to suffix "1803" to the current Win10 worker
> name?
>
> I'll submit the PR for the new worker once there is clarification.
>
> Thanks for your time.
>
> --
> Jeremy Kloth
> ___
> Python-Buildbots mailing list -- python-buildbots@python.org
> To unsubscribe send an email to python-buildbots-le...@python.org
> https://mail.python.org/mailman3/lists/python-buildbots.python.org/
> Member address: db3l@gmail.com
>
___
Python-Buildbots mailing list -- python-buildbots@python.org
To unsubscribe send an email to python-buildbots-le...@python.org
https://mail.python.org/mailman3/lists/python-buildbots.python.org/
Member address: arch...@mail-archive.com

[Python-buildbots] Re: buildmaster worker naming

[David Bolen]

On Mon, Mar 29, 2021 at 12:58 PM Jeremy Kloth 
wrote:

> While I appreciate the information, I believe that the names being
> referred to here are the worker IDs.  I'm wondering about the worker
> *display* names, those that are shown in the various status views.  In
> the buildmaster config, this name forms the base of the builder names.
>

Ah, yes, for my part you're correct we were referring to different things,
probably due to the original use of worker, rather than builder (though the
suggested text ought to have suggested builder to me).

For myself I rarely consider builder names.  My most common landing page is
the workers summary page, then passing through the all builds summary for a
worker or an individual builder summary page on the way to a specific
build.  In those cases the builder name is mostly noise to skip past when
looking for the branch.  I no longer have multiple builders on any worker,
but when I did I only needed uniqueness within the worker.  Of course,
users of other status pages (I can't recall when I last used the waterfall)
may have different needs, though they can always drill through to the
worker for worker-level details.

While I suppose the above argues for me not to care much about builder
names or updates, it's then a more pragmatic issue of ease of my updating
details in sync with the worker, for which the info file is simpler.  It
also feels more appropriate as a worker level detail rather than builder,
though I'd agree Windows 10 and its feature upgrades complicates this point.

-- David
___
Python-Buildbots mailing list -- python-buildbots@python.org
To unsubscribe send an email to python-buildbots-le...@python.org
https://mail.python.org/mailman3/lists/python-buildbots.python.org/
Member address: arch...@mail-archive.com

[Python-buildbots] Paused workers

[David Bolen]

There seem to be a handful of workers who have been in a "paused" state for
the past 24 hours or so, including my Win10 worker.  Does anyone know what
might be the cause?  I experimented with trying to resume mine, but don't
have the necessary admin rights.  The worker appeared to be working
normally right up until it was paused.

-- David
___
Python-Buildbots mailing list -- python-buildbots@python.org
To unsubscribe send an email to python-buildbots-le...@python.org
https://mail.python.org/mailman3/lists/python-buildbots.python.org/
Member address: arch...@mail-archive.com

[Python-buildbots] Re: Paused workers

[David Bolen]

Right now I don't see any other paused workers (from the State column on
the workers page).

My worker (bolen-windows10) resumed Monday evening - at the time I believe
it was the only remaining paused worker.  I assumed someone had unpaused
it, but if not, perhaps it's something the master can do on its own.  If
so, maybe there are logs on the master that could identify a reason.

As for the feature, I'm not really familiar with it either, but "Pause" and
"Unpause" appear to be two of the actions that can be performed against a
worker (actions button on the workers page or clicking the state icon next
a worker), though executing them requires admin rights.  Once paused, all
jobs to a worker appear to fail with an exception about having no builders
(which you can see on any of the "AMD64 Windows 10" builds, on any branch,
a few days back) and the worker shows a pause icon next to it as its state
on the workers page or on its builds page.

-- David

On Wed, Aug 4, 2021 at 11:13 AM Victor Stinner  wrote:

> Hi,
>
> I don't know what is this "pause" feature", what can cause a pause,
> nor how to resume a buildbot worker.
>
> "AMD64 Windows10 3.x" and "AMD64 Windows10 Pro 3.x" built Python
> successfully 5 hours ago.
>
> Does someone know if any other worker is paused?
>
> Victor
>
> On Thu, Jul 29, 2021 at 10:03 PM David Bolen  wrote:
> >
> > There seem to be a handful of workers who have been in a "paused" state
> for
> > the past 24 hours or so, including my Win10 worker.  Does anyone know
> what
> > might be the cause?  I experimented with trying to resume mine, but don't
> > have the necessary admin rights.  The worker appeared to be working
> > normally right up until it was paused.
> >
> > -- David
> > ___
> > Python-Buildbots mailing list -- python-buildbots@python.org
> > To unsubscribe send an email to python-buildbots-le...@python.org
> > https://mail.python.org/mailman3/lists/python-buildbots.python.org/
> > Member address: vstin...@python.org
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
>
___
Python-Buildbots mailing list -- python-buildbots@python.org
To unsubscribe send an email to python-buildbots-le...@python.org
https://mail.python.org/mailman3/lists/python-buildbots.python.org/
Member address: arch...@mail-archive.com