Hi Jay, Which version of Icinga Web 2 are you using?
Best, Eric On 19 Jan 2016, at 2:33 PM, Jay Newman <jay.new...@esentire.com<mailto:jay.new...@esentire.com>> wrote: My apologies if this ends up as a duplicate, but I believe I used the wrong address on my first attempt. Hello all, Our team reported Icinga as down, and that is how it displayed in the status of the icingaweb2 interface, though at a command prompt the service was still running. Curious as to whether it could be a problem with the database being down or slow, I checked the mysql server and found it had 70 processes running – which seems rather high. Text included further below as sample output. I stopped the Icinga service, waited a minute, and expected the process count to drop to a very low number, but this was not the case. Something on the Icinga server was still making new connections as the process count was now 73. Eventually, I stopped Icinga, restarted mysql, then started Icinga. After a few minutes status was normal. This is not something we want to go through on a daily basis, but it has happened several times this week. Any suggestions or advice as to how to remedy / stabilize the situation? I have also pasted the performance information so you have an idea of the size of the environment and how it typically runs. icinga2 --version icinga2 - The Icinga 2 network monitoring daemon (version: r2.4.1-1) System information: Platform: Debian GNU/Linux Platform version: 14.04 Kernel: Linux Kernel version: 3.13.0-39-generic Architecture: x86_64 Regards, Jay Newman, CDCDP eSentire Inc. Performance Info Object summaries overall scheduled Hosts 1388 1388 Services 43148 43148 Average services per host 31.09 31.09 Active checks Latency Execution time Host Checks 1388 0.000s 1.368s Service Checks 43148 0.000s 2.638s Passive checks Host Checks 0 Service Checks 0 mysql> show processlist; +--------+-------------+-------------------------------+-----------------+---------+-------+-----------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+ | Id | User | Host | db | Command | Time | State | Info | +--------+-------------+-------------------------------+-----------------+---------+-------+-----------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+ | 2 | system user | | NULL | Connect | 3780 | Slave has read all relay log; waiting for the slave I/O thread to update it | NULL | | 544460 | icinga | master1.icinga.internal:50828 | icinga2idomysql | Query | 80382 | Sending data | SELECT eho.*, eho.name1 COLLATE latin1_general_ci AS host_name, eho.name2 COLLATE latin1_general_ci | | 551814 | icinga | master1.icinga.internal:58370 | icinga2idomysql | Query | 77906 | Sending data | SELECT eho.*, eho.name1 COLLATE latin1_general_ci AS host_name, eho.name2 COLLATE latin1_general_ci | | 559920 | icinga | master1.icinga.internal:38462 | icinga2idomysql | Query | 75370 | Sending data | SELECT eho.*, eho.name1 COLLATE latin1_general_ci AS host_name, eho.name2 COLLATE latin1_general_ci | | 674544 | root | 10.1.170.31:51661 | NULL | Sleep | 549 | | NULL | | 674554 | root | 10.1.170.31:51663 | NULL | Sleep | 549 | | NULL | | 797108 | icinga | master1.icinga.internal:58961 | icinga2idomysql | Killed | 2607 | NULL | NULL | | 798244 | icinga | master1.icinga.internal:60218 | icinga2idomysql | Query | 2462 | Sending data | SELECT eho.*, eho.name1 COLLATE latin1_general_ci AS host_name, eho.name2 COLLATE latin1_general_ci | | 800313 | icinga | master1.icinga.internal:34574 | icinga2idomysql | Query | 651 | updating | DELETE FROM icinga_comments WHERE instance_id = 1 | | 800370 | icinga | master1.icinga.internal:34639 | icinga2idomysql | Query | 629 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800371 | icinga | master1.icinga.internal:34640 | icinga2idomysql | Query | 629 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800372 | icinga | master1.icinga.internal:34641 | icinga2idomysql | Query | 629 | Sending data | SELECT SUM(CASE WHEN object_type = 'service' THEN 1 ELSE 0 END) AS services_total, SUM(CASE WHEN obj | | 800373 | icinga | master1.icinga.internal:34642 | icinga2idomysql | Query | 629 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800374 | icinga | master1.icinga.internal:34643 | icinga2idomysql | Query | 629 | Sending data | SELECT SUM(CASE WHEN object_type = 'service' THEN 1 ELSE 0 END) AS services_total, SUM(CASE WHEN obj | | 800375 | icinga | master1.icinga.internal:34644 | icinga2idomysql | Query | 628 | Copying to tmp table | SELECT hs.* FROM (SELECT CASE WHEN ss.active_checks_enabled = 0 AND ss.passive_checks_enabled = 1 TH | | 800376 | icinga | master1.icinga.internal:34645 | icinga2idomysql | Query | 628 | Copying to tmp table | SELECT hs.* FROM (SELECT CASE WHEN ss.active_checks_enabled = 0 AND ss.passive_checks_enabled = 1 TH | | 800377 | icinga | master1.icinga.internal:34646 | icinga2idomysql | Query | 628 | Copying to tmp table | SELECT hs.* FROM (SELECT CASE WHEN ss.active_checks_enabled = 0 AND ss.passive_checks_enabled = 1 TH | | 800378 | icinga | master1.icinga.internal:34647 | icinga2idomysql | Query | 628 | Copying to tmp table | SELECT cm.internal_comment_id AS id, UNIX_TIMESTAMP(cm.comment_time) AS timestamp, cm.author_name CO | | 800379 | icinga | master1.icinga.internal:34648 | icinga2idomysql | Query | 627 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800380 | icinga | master1.icinga.internal:34649 | icinga2idomysql | Query | 627 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800381 | icinga | master1.icinga.internal:34650 | icinga2idomysql | Query | 627 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800382 | icinga | master1.icinga.internal:34651 | icinga2idomysql | Query | 627 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800383 | icinga | master1.icinga.internal:34652 | icinga2idomysql | Query | 627 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800384 | icinga | master1.icinga.internal:34653 | icinga2idomysql | Query | 626 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800385 | icinga | master1.icinga.internal:34654 | icinga2idomysql | Query | 626 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800386 | icinga | master1.icinga.internal:34655 | icinga2idomysql | Query | 622 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800387 | icinga | master1.icinga.internal:34656 | icinga2idomysql | Query | 622 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800388 | icinga | master1.icinga.internal:34657 | icinga2idomysql | Query | 622 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800389 | icinga | master1.icinga.internal:34658 | icinga2idomysql | Query | 622 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800390 | icinga | master1.icinga.internal:34659 | icinga2idomysql | Query | 621 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800391 | icinga | master1.icinga.internal:34660 | icinga2idomysql | Query | 621 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800392 | icinga | master1.icinga.internal:34661 | icinga2idomysql | Query | 615 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800393 | icinga | master1.icinga.internal:34662 | icinga2idomysql | Query | 614 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800394 | icinga | master1.icinga.internal:34663 | icinga2idomysql | Query | 614 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800395 | icinga | master1.icinga.internal:34664 | icinga2idomysql | Query | 613 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800396 | icinga | master1.icinga.internal:34665 | icinga2idomysql | Query | 613 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800397 | icinga | master1.icinga.internal:34666 | icinga2idomysql | Query | 567 | Copying to tmp table | SELECT hs.* FROM (SELECT CASE WHEN ss.active_checks_enabled = 0 AND ss.passive_checks_enabled = 1 TH | | 800398 | icinga | master1.icinga.internal:34668 | icinga2idomysql | Query | 535 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800399 | icinga | master1.icinga.internal:34669 | icinga2idomysql | Query | 535 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800400 | icinga | master1.icinga.internal:34670 | icinga2idomysql | Query | 535 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800401 | icinga | master1.icinga.internal:34671 | icinga2idomysql | Query | 535 | Sending data | SELECT SUM(CASE WHEN object_type = 'host' AND state = 1 AND acknowledged + in_downtime = 0 THEN 1 EL | | 800406 | icinga | master1.icinga.internal:34676 | icinga2idomysql | Query | 528 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800407 | icinga | master1.icinga.internal:34677 | icinga2idomysql | Query | 528 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800408 | icinga | master1.icinga.internal:34678 | icinga2idomysql | Query | 528 | Sending data | SELECT COUNT(*) AS cnt FROM (SELECT statussummary.hostgroup, statussummary.hostgroup_alias, SUM(CASE | | 800409 | icinga | master1.icinga.internal:34679 | icinga2idomysql | Query | 528 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM (SELECT statussummary.servicegroup, statussummary.servicegroup_alias, SU | | 800410 | icinga | master1.icinga.internal:34680 | icinga2idomysql | Query | 528 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800411 | icinga | master1.icinga.internal:34681 | icinga2idomysql | Query | 528 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM (SELECT statussummary.servicegroup, statussummary.servicegroup_alias, SU | | 800412 | icinga | master1.icinga.internal:34682 | icinga2idomysql | Query | 528 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800413 | icinga | master1.icinga.internal:34683 | icinga2idomysql | Query | 528 | Sending data | SELECT COUNT(*) AS cnt FROM (SELECT statussummary.hostgroup, statussummary.hostgroup_alias, SUM(CASE | | 800414 | icinga | master1.icinga.internal:34684 | icinga2idomysql | Query | 528 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800415 | icinga | master1.icinga.internal:34685 | icinga2idomysql | Query | 527 | Sending data | SELECT COUNT(*) AS cnt FROM (SELECT statussummary.hostgroup, statussummary.hostgroup_alias, SUM(CASE | | 800416 | icinga | master1.icinga.internal:34686 | icinga2idomysql | Query | 527 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800417 | icinga | master1.icinga.internal:34687 | icinga2idomysql | Query | 526 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800418 | icinga | master1.icinga.internal:34688 | icinga2idomysql | Query | 526 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800419 | icinga | master1.icinga.internal:34690 | icinga2idomysql | Query | 526 | Sending data | SELECT COUNT(*) AS cnt FROM (SELECT statussummary.hostgroup, statussummary.hostgroup_alias, SUM(CASE | | 800420 | icinga | master1.icinga.internal:34689 | icinga2idomysql | Query | 526 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM (SELECT statussummary.servicegroup, statussummary.servicegroup_alias, SU | | 800421 | icinga | master1.icinga.internal:34693 | icinga2idomysql | Query | 282 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800422 | icinga | master1.icinga.internal:34694 | icinga2idomysql | Query | 282 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800423 | icinga | master1.icinga.internal:34695 | icinga2idomysql | Query | 282 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800424 | icinga | master1.icinga.internal:34696 | icinga2idomysql | Query | 278 | Copying to tmp table | SELECT hs.* FROM (SELECT CASE WHEN ss.active_checks_enabled = 0 AND ss.passive_checks_enabled = 1 TH | | 800425 | icinga | master1.icinga.internal:34697 | icinga2idomysql | Query | 273 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800426 | icinga | master1.icinga.internal:34698 | icinga2idomysql | Query | 273 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800427 | icinga | master1.icinga.internal:34699 | icinga2idomysql | Query | 273 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800428 | icinga | master1.icinga.internal:34700 | icinga2idomysql | Query | 272 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800429 | icinga | master1.icinga.internal:34701 | icinga2idomysql | Query | 271 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800430 | icinga | master1.icinga.internal:34702 | icinga2idomysql | Query | 271 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800431 | icinga | master1.icinga.internal:34703 | icinga2idomysql | Query | 271 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800432 | icinga | master1.icinga.internal:34704 | icinga2idomysql | Query | 29 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800433 | icinga | master1.icinga.internal:34705 | icinga2idomysql | Query | 29 | Copying to tmp table | SELECT COUNT(*) AS cnt FROM icinga_objects AS ho INNER JOIN icinga_hoststatus AS hs ON ho.object_id | | 800434 | root | localhost | NULL | Query | 0 | NULL | show processlist | +--------+-------------+-------------------------------+-----------------+---------+-------+-----------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+ 70 rows in set (0.01 sec) _______________________________________________ icinga-users mailing list icinga-users@lists.icinga.org<mailto:icinga-users@lists.icinga.org> https://lists.icinga.org/mailman/listinfo/icinga-users -- Eric Lippmann Lead Senior Developer NETWAYS GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB18461 http://www.netways.de | eric.lippm...@netways.de ** OSDC 2016 - April - netways.de/osdc ** ** OSBConf 2016 - September - osbconf.org ** _______________________________________________ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
