Ian, thanks for digging in and helping sort out some of these issues!
Ian Wienand <mailto:iwien...@redhat.com>
April 4, 2018 at 11:04 PM
We've long had problems with this host and I've looked at it before
[1]. It often drops out.
It seems there's enough interest we should dive a bit deeper. Here's
what I've found out:
askbot
------
Of the askbot site, it seems under control, except for an unbounded
session log file. Proposed [2]
root@ask:/srv# du -hs *
2.0G askbot-site
579M dist
overall
-------
The major consumer is /var; where we've got
3.9G log
5.9G backups
9.4G lib
backups
-------
The backup seem under control at least; we're rotating them out and we
keep 10, and the size is pretty consistently 500mb:
root@ask:/var/backups/pgsql_backups# ls -lh
total 5.9G
-rw-r--r-- 1 root root 599M Apr 5 00:03 askbotdb.sql.gz
-rw-r--r-- 1 root root 598M Apr 4 00:03 askbotdb.sql.gz.1
...
We could reduce the backup rotations to just one if we like -- the
server is backed up nightly via bup, so at any point we can get
previous dumps from there. bup should de-duplicate everything, but
still, it's probably not necessary.
The db directory was sitting at ~9gb
root@ask:/var/lib/postgresql# du -hs
8.9G .
AFAICT, it seems like the autovacuum is running OK on the busy tables
askbotdb=# select relname,last_vacuum, last_autovacuum, last_analyze,
last_autoanalyze from pg_stat_user_tables where last_autovacuum is not
NULL;
relname | last_vacuum | last_autovacuum | last_analyze | last_autoanalyze
------------------+-------------+-------------------------------+-------------------------------+-------------------------------
django_session | | 2018-04-02 17:29:48.329915+00 | 2018-04-05
02:18:39.300126+00 | 2018-04-05 00:11:23.456602+00
askbot_badgedata | | 2018-04-04 07:19:21.357461+00 | | 2018-04-04
07:18:16.201376+00
askbot_thread | | 2018-04-04 16:24:45.124492+00 | | 2018-04-04
20:32:25.845164+00
auth_message | | 2018-04-04 12:29:24.273651+00 | 2018-04-05
02:18:07.633781+00 | 2018-04-04 21:26:38.178586+00
djkombu_message | | 2018-04-05 02:11:50.186631+00 | | 2018-04-05
02:14:45.22926+00
Out of interest I did run a manual
su - postgres -c "vacuumdb --all --full --analyze"
We dropped something
root@ask:/var/lib/postgresql# du -hs
8.9G .
(after)
5.8G
I installed pg_activity and watched for a while; nothing seemed to be
really stressing it.
Ergo, I'm not sure if there's much to do in the db layers.
logs
----
This leaves the logs
1.1G jetty
2.9G apache2
The jetty logs are cleaned regularly. I think they could be made more
quiet, but they seem to be bounded.
Apache logs are rotated but never cleaned up. Surely logs from 2015
aren't useful. Proposed [3]
Random offline
--------------
[3] is an example of a user reporting the site was offline. Looking
at the logs, it seems that puppet found httpd not running at 07:14 and
restarted it:
Apr 4 07:14:40 ask puppet-user[20737]:
(Scope(Class[Postgresql::Server])) Passing "version" to
postgresql::server is deprecated; please use postgresql::globals instead.
Apr 4 07:14:42 ask puppet-user[20737]: Compiled catalog for
ask.openstack.org in environment production in 4.59 seconds
Apr 4 07:14:44 ask crontab[20987]: (root) LIST (root)
Apr 4 07:14:49 ask puppet-user[20737]:
(/Stage[main]/Httpd/Service[httpd]/ensure) ensure changed 'stopped' to
'running'
Apr 4 07:14:54 ask puppet-user[20737]: Finished catalog run in 10.43
seconds
Which first explains why when I looked, it seemed OK. Checking the
apache logs we have:
[Wed Apr 04 07:01:08.144746 2018] [:error] [pid 12491:tid
140439253419776] [remote 176.233.126.142:43414] mod_wsgi (pid=12491):
Exception occurred processing WSGI script
'/srv/askbot-site/config/django.wsgi'.
[Wed Apr 04 07:01:08.144870 2018] [:error] [pid 12491:tid
140439253419776] [remote 176.233.126.142:43414] IOError: failed to
write data
... more until ...
[Wed Apr 04 07:15:58.270180 2018] [:error] [pid 17060:tid
140439253419776] [remote 176.233.126.142:43414] mod_wsgi (pid=17060):
Exception occurred processing WSGI script
'/srv/askbot-site/config/django.wsgi'.
[Wed Apr 04 07:15:58.270303 2018] [:error] [pid 17060:tid
140439253419776] [remote 176.233.126.142:43414] IOError: failed to
write data
and the restart logged
[Wed Apr 04 07:14:48.912626 2018] [core:warn] [pid 21247:tid
140439370192768] AH00098: pid file /var/run/apache2/apache2.pid
overwritten -- Unclean shutdown of previous Apache run?
[Wed Apr 04 07:14:48.913548 2018] [mpm_event:notice] [pid 21247:tid
140439370192768] AH00489: Apache/2.4.7 (Ubuntu) OpenSSL/1.0.1f
mod_wsgi/3.4 Python/2.7.6 configured -- resuming normal operations
[Wed Apr 04 07:14:48.913583 2018] [core:notice] [pid 21247:tid
140439370192768] AH00094: Command line: '/usr/sbin/apache2'
[Wed Apr 04 14:59:55.408060 2018] [mpm_event:error] [pid 21247:tid
140439370192768] AH00485: scoreboard is full, not at MaxRequestWorkers
This does not appear to be disk-space related; see the cacti graphs
for that period that show the disk is full-ish, but not full [5].
What caused the I/O errors? dmesg has nothing in it since 30/Mar.
kern.log is empty.
Server
------
Most importantly, this sever wants a Xenial upgrade. At the very
least that apache is known to handle the "scoreboard is full" issue
better.
We should ensure that we use a bigger instance; it's using up some
swap
postgres@ask:~$ free -h
total used free shared buffers cached
Mem: 3.9G 3.6G 269M 136M 11M 819M
-/+ buffers/cache: 2.8G 1.1G
Swap: 3.8G 259M 3.6G
tl;dr
-----
I don't think there's anything run-away bad going on, but the server
is undersized and needs a system update.
Since I've got this far with it, over the next few days I'll see where
we are with the puppet for a Xenial upgrade and see if we can't get a
migration underway.
Thanks,
-i
[1] https://review.openstack.org/406670
[2] https://review.openstack.org/558977
[3] https://review.openstack.org/558985
[4]
http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2018-04-04.log.html#t2018-04-04T07:11:22
[5]
http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=2547&rra_id=0&view_type=tree&graph_start=1522859103&graph_end=1522879839
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Paul Belanger <mailto:pabelan...@redhat.com>
April 4, 2018 at 5:30 PM
We also have a 2nd issue where the ask.o.o server doesn't appear to be
large
enough any more to handle the traffic. A few times over the last few
weeks we've
had outages due to the HDD being full.
We likely need to reduce the number of days we retain database backups
/ http
logs or look to attach a volume to increase storage.
Paul
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Jimmy McArthur <mailto:ji...@openstack.org>
April 4, 2018 at 4:26 PM
Hi everyone!
We have a very robust and vibrant community at ask.openstack.org
<https://ask.openstack.org/>. There are literally dozens of posts a
day. However, many of them don't receive knowledgeable answers. I'm
really worried about this becoming a vacuum where potential community
members get frustrated and don't realize how to get more involved with
the community.
I'm looking for thoughts/ideas/feelings about this tool as well as
potential admin volunteers to help us manage the constant influx of
technical and not-so-technical questions around OpenStack.
For those of you already contributing there, Thank You! For those
that are interested in becoming a moderator (instant AUC status!) or
have some additional ideas around fostering this community, please
respond.
Looking forward to your thoughts :)
Thanks!
Jimmy
irc: jamesmcarthur
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
