Re: [OpenStack-Infra] pypi volume downtime

2016-12-05 Thread Ian Wienand
On 6 Dec. 2016 3:13 am, "Kevin L. Mitchell" wrote:On Mon, 2016-12-05 at 15:30 +1100, Ian Wienand wrote: For the record, those log entries are from December 2nd, rather than February: US date conventions. Heh, yep :). In one of the openafs files it has at the top /* 1/1/89: NB: this stuff is all g

Re: [OpenStack-Infra] pypi volume downtime

2016-12-05 Thread Jeremy Stanley
On 2016-12-05 10:10:22 -0600 (-0600), Kevin L. Mitchell wrote: > On Mon, 2016-12-05 at 15:30 +1100, Ian Wienand wrote: > > As for the root cause, I don't see anything else particularly > > insightful in the logs. The salvage server logs, implicated above, > > end in Feburary which isn't very helpf

Re: [OpenStack-Infra] pypi volume downtime

2016-12-05 Thread Kevin L. Mitchell
On Mon, 2016-12-05 at 15:30 +1100, Ian Wienand wrote: > As for the root cause, I don't see anything else particularly > insightful in the logs. The salvage server logs, implicated above, > end in Feburary which isn't very helpful > > --- SalsrvLog.old --- > 12/02/2016 04:19:59 SALVAGING VOLUME 5

Re: [OpenStack-Infra] pypi volume downtime

2016-12-04 Thread Ian Wienand
On 12/05/2016 03:30 PM, Ian Wienand wrote: So I think the only side-effect at the moment is that while the bandersnatch cron update is running, AFS is locked and thus the mirrors will not get a new volume release until this sync is done; i.e. our pypi mirrors are a bit behind. As of right now,

[OpenStack-Infra] pypi volume downtime

2016-12-04 Thread Ian Wienand
Hi, Today I was alerted to jobs failing on IRC, further investigation showed the pypi volume did not seem to be responding on the mirror servers. --- ianw@mirror:/afs/openstack.org/mirror$ ls pypi ls: cannot access pypi: Connection timed out --- The bandersnatch logs suggested the vos release