Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Paul Belanger
On Tue, Mar 07, 2017 at 07:39:13PM +1100, Ian Wienand wrote: > On 03/07/2017 07:20 PM, Gene Kuo wrote: > > These errors do line up to the time where it's down. > > However, I have no idea what cause apache to seg fault. > > Something disappearing underneath it would be my suspicion > > Anyway, I

Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Ian Wienand
On 03/07/2017 07:20 PM, Gene Kuo wrote: These errors do line up to the time where it's down. However, I have no idea what cause apache to seg fault. Something disappearing underneath it would be my suspicion Anyway, I added "CoreDumpDirectory /var/cache/apache2" to /etc/apache2/apache2.conf ma

Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Gene Kuo
These errors do line up to the time where it's down. However, I have no idea what cause apache to seg fault. -Original Message- From: "Ian Wienand" Sent: Tuesday, March 7, 2017 3:10am To: "Gene Kuo" , "Tom Fifield" Cc: openstack-infra@lists.openstac

Re: [OpenStack-Infra] Ask.o.o down

2017-03-07 Thread Ian Wienand
On 03/07/2017 06:46 PM, Gene Kuo wrote: > I found that ask.o.o is down again. I restarted apache --- root@ask:/var/log/apache2# date Tue Mar 7 07:54:26 UTC 2017 [Tue Mar 07 06:01:38.469993 2017] [core:notice] [pid 19511:tid 140460060575616] AH00052: child pid 19517 exit signal Segmentation fa

Re: [OpenStack-Infra] Ask.o.o down

2017-03-06 Thread Gene Kuo
uot; Sent: Tuesday, February 21, 2017 2:15am To: openstack-infra@lists.openstack.org Subject: Re: [OpenStack-Infra] Ask.o.o down On 廿十七年二月廿一日 暮 03:11, Tom Fifield wrote: > > > On 廿十七年二月十四日 暮 04:19, Joshua Hesketh wrote: >> >> >> On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield

Re: [OpenStack-Infra] Ask.o.o down

2017-02-20 Thread Tom Fifield
On 廿十七年二月廿一日 暮 03:11, Tom Fifield wrote: On 廿十七年二月十四日 暮 04:19, Joshua Hesketh wrote: On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield mailto:t...@openstack.org>> wrote: On 14/02/17 16:11, Joshua Hesketh wrote: Hey Tom, Where is that script being fired from (a quick grep

Re: [OpenStack-Infra] Ask.o.o down

2017-02-20 Thread Tom Fifield
On 廿十七年二月十四日 暮 04:19, Joshua Hesketh wrote: On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield mailto:t...@openstack.org>> wrote: On 14/02/17 16:11, Joshua Hesketh wrote: Hey Tom, Where is that script being fired from (a quick grep doesn't find it), or is it a

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Joshua Hesketh
On Tue, Feb 14, 2017 at 7:15 PM, Tom Fifield wrote: > On 14/02/17 16:11, Joshua Hesketh wrote: > >> Hey Tom, >> >> Where is that script being fired from (a quick grep doesn't find it), or >> is it a tool people are using? >> >> If it's a tool we'd need to make sure whoever is using it gets a new

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Tom Fifield
On 14/02/17 16:11, Joshua Hesketh wrote: Hey Tom, Where is that script being fired from (a quick grep doesn't find it), or is it a tool people are using? If it's a tool we'd need to make sure whoever is using it gets a new version to rule it out. Indeed. It's fired from a PHP service on www

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Joshua Hesketh
Hey Tom, Where is that script being fired from (a quick grep doesn't find it), or is it a tool people are using? If it's a tool we'd need to make sure whoever is using it gets a new version to rule it out. - Josh On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield wrote: > On 14/02/17 16:06, Joshua

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Tom Fifield
On 14/02/17 16:06, Joshua Hesketh wrote: Hey, I've brought the service back up, but have no new clues as to why. Cheers. Going to try: https://review.openstack.org/#/c/433478/ to see if this script is culprit. - Josh On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield mailto:t...@openstack.org>>

Re: [OpenStack-Infra] Ask.o.o down

2017-02-14 Thread Joshua Hesketh
Hey, I've brought the service back up, but have no new clues as to why. - Josh On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield wrote: > On 10/02/17 22:39, Jeremy Stanley wrote: > >> On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote: >> [...] >> >>> Down again, this time with "Network is unr

Re: [OpenStack-Infra] Ask.o.o down

2017-02-13 Thread Tom Fifield
On 10/02/17 22:39, Jeremy Stanley wrote: On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote: [...] Down again, this time with "Network is unreachable". [...] I'm not finding any obvious errors on the server nor relevant maintenance notices/trouble tickets from the service provider to expl

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Jeremy Stanley
On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote: [...] > Down again, this time with "Network is unreachable". [...] I'm not finding any obvious errors on the server nor relevant maintenance notices/trouble tickets from the service provider to explain this. I do see conspicuous gaps in netw

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Tom Fifield
On 10/02/17 16:41, Marton Kiss wrote: Hi Tom Still not working from your end? Works perfectly from here. It is back now. Marton On Fri, Feb 10, 2017 at 9:14 AM Tom Fifield mailto:t...@openstack.org>> wrote: On 10/02/17 16:08, Tom Fifield wrote: > On 13/01/17 14:41, Tom Fifield wro

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Marton Kiss
Hi Tom Still not working from your end? Works perfectly from here. Marton On Fri, Feb 10, 2017 at 9:14 AM Tom Fifield wrote: > On 10/02/17 16:08, Tom Fifield wrote: > > On 13/01/17 14:41, Tom Fifield wrote: > >> As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on > >> IPv4) -

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Tom Fifield
On 13/01/17 14:41, Tom Fifield wrote: As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on IPv4) - tried from a couple of different hosts fifieldt@docwork2:~$ wget ask.openstack.org --2017-01-13 06:40:49-- http://ask.openstack.org/ Resolving ask.openstack.org (ask.openstack.org)

Re: [OpenStack-Infra] Ask.o.o down

2017-02-10 Thread Tom Fifield
On 10/02/17 16:08, Tom Fifield wrote: On 13/01/17 14:41, Tom Fifield wrote: As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on IPv4) - tried from a couple of different hosts fifieldt@docwork2:~$ wget ask.openstack.org --2017-01-13 06:40:49-- http://ask.openstack.org/ Resolvin

Re: [OpenStack-Infra] Ask.o.o down

2017-01-13 Thread Jeremy Stanley
On 2017-01-13 10:33:24 + (+), Marton Kiss wrote: > You can find more details about the host here: > http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=156 > It had a network outage somewhere, if you check the eth0, the > traffic was zero. Unfortunately I find no

Re: [OpenStack-Infra] Ask.o.o down

2017-01-13 Thread Marton Kiss
Tom, You can find more details about the host here: http://cacti.openstack.org/cacti/graph_view.php?action=tree&tree_id=1&leaf_id=156 It had a network outage somewhere, if you check the eth0, the traffic was zero. Marton On Fri, Jan 13, 2017 at 8:05 AM Tom Fifield wrote: > ... and at 0700Z it

Re: [OpenStack-Infra] Ask.o.o down

2017-01-12 Thread Tom Fifield
... and at 0700Z it is back ... On 13/01/17 14:41, Tom Fifield wrote: As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on IPv4) - tried from a couple of different hosts fifieldt@docwork2:~$ wget ask.openstack.org --2017-01-13 06:40:49-- http://ask.openstack.org/ Resolving ask.

[OpenStack-Infra] Ask.o.o down

2017-01-12 Thread Tom Fifield
As at 2017-01-13 0641Z, Ask.o.o is refusing connections (at least on IPv4) - tried from a couple of different hosts fifieldt@docwork2:~$ wget ask.openstack.org --2017-01-13 06:40:49-- http://ask.openstack.org/ Resolving ask.openstack.org (ask.openstack.org)... 23.253.72.95, 2001:4800:7815:103: