Re: [OpenStack-Infra] Gate Issues

2017-12-08 Thread David Shrewsbury
The locked nodes were due to something Clark found earlier this week, and hopefully fixed with: https://review.openstack.org/526234 Short story is that the request handlers (holding the locks on the nodes), were never being allowed to continue processing because of an exception being thrown that

Re: [OpenStack-Infra] Gate Issues

2017-12-08 Thread Paul Belanger
On Fri, Dec 08, 2017 at 08:38:24PM +1100, Ian Wienand wrote: > Hello, > > Just to save people reverse-engineering IRC logs... > > At ~04:00UTC frickler called out that things had been sitting in the > gate for ~17 hours. > > Upon investigation, one of the stuck jobs was a > legacy-tempest-dsvm-n

Re: [OpenStack-Infra] Gate Issues

2017-12-08 Thread Paul Belanger
On Fri, Dec 08, 2017 at 08:56:58PM +1100, Ian Wienand wrote: > On 12/08/2017 08:38 PM, Ian Wienand wrote: > > However, the gate did not become healthy. Upon further investigation, > > the executors are very frequently failing jobs with > > > > 2017-12-08 06:41:10,412 ERROR zuul.AnsibleJob: [bui

Re: [OpenStack-Infra] Gate Issues

2017-12-08 Thread Ian Wienand
On 12/08/2017 08:38 PM, Ian Wienand wrote: However, the gate did not become healthy. Upon further investigation, the executors are very frequently failing jobs with 2017-12-08 06:41:10,412 ERROR zuul.AnsibleJob: [build: 11062f1cca144052afb733813cdb16d8] Exception while executing job Traceb