On 14/02/17 16:11, Joshua Hesketh wrote:
Hey Tom,
Where is that script being fired from (a quick grep doesn't find it), or
is it a tool people are using?
If it's a tool we'd need to make sure whoever is using it gets a new
version to rule it out.
Indeed.
It's fired from a PHP service on www.openstack.org itself, which writes
to the Member database:
https://github.com/OpenStackweb/openstack-org/blob/master/auc-metrics/code/services/ActiveModeratorService.php
The next step is to update the copy of the script it references:
https://github.com/OpenStackweb/openstack-org/blob/master/auc-metrics/lib/uc-recognition/tools/get_active_moderator.py
I am not sure if this is in place using git submodules or manually, but
will figure it out and get that updated.
- Josh
On Tue, Feb 14, 2017 at 7:07 PM, Tom Fifield <t...@openstack.org
<mailto:t...@openstack.org>> wrote:
On 14/02/17 16:06, Joshua Hesketh wrote:
Hey,
I've brought the service back up, but have no new clues as to why.
Cheers.
Going to try: https://review.openstack.org/#/c/433478/
<https://review.openstack.org/#/c/433478/>
to see if this script is culprit.
- Josh
On Tue, Feb 14, 2017 at 6:50 PM, Tom Fifield <t...@openstack.org
<mailto:t...@openstack.org>
<mailto:t...@openstack.org <mailto:t...@openstack.org>>> wrote:
On 10/02/17 22:39, Jeremy Stanley wrote:
On 2017-02-10 16:08:51 +0800 (+0800), Tom Fifield wrote:
[...]
Down again, this time with "Network is unreachable".
[...]
I'm not finding any obvious errors on the server nor
relevant
maintenance notices/trouble tickets from the service
provider to
explain this. I do see conspicuous gaps in network
traffic volume
and system load from ~06:45 to ~08:10 UTC according to
cacti:
http://cacti.openstack.org/?tree_id=1&leaf_id=156
<http://cacti.openstack.org/?tree_id=1&leaf_id=156>
<http://cacti.openstack.org/?tree_id=1&leaf_id=156
<http://cacti.openstack.org/?tree_id=1&leaf_id=156>>
Skipping back through previous days I find some similar gaps
starting anywhere from 06:30 to 07:00 and ending between
07:00 and
08:00 but they don't seem to occur every day and I'm not
having much
luck finding a pattern. It _is_ conspicuously close to when
/etc/cron.daily scripts get fired from the crontab so
might coincide
with log rotation/service restarts? The graphs don't
show these gaps
correlating with any spikes in CPU, memory or disk
activity so it
doesn't seem to be resource starvation (at least not for
any common
resources we're tracking).
Indeed. It's down again today during the same timeslot.
Another idea for the cron-based theory:
https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py
<https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py>
<https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py
<https://github.com/openstack/uc-recognition/blob/master/tools/get_active_moderator.py>>
loops through the list of Ask OpenStack users via the API on
a cron
running on www.openstack.org <http://www.openstack.org>
<http://www.openstack.org>. Not sure
when that cron runs, but if it's similar, this could
potentially be
a high-load generator.
Regards,
Tom
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>
<mailto:OpenStack-Infra@lists.openstack.org
<mailto:OpenStack-Infra@lists.openstack.org>>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra>>
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
