Dear Icinga experts, in CERN ATLAS experiment we setup our Icinga2 monitoring system using two nodes configured in High Availability. The Icinga2 version used is v2.3.11 (see below for more details) and we do have ido-mysql enabled. The server is checking 4141 hosts and 83188 services.
We tried few months ago to upgrade to v2.4.1, unfortunately the restart of the icinga2 service was hunging forever apparently due to some bad interaction with mysql. Once we disabled ido-mysql, it started. We setup a test server, identical to the production one with the exception of the HA, where to be able to play and we observed the same behaviour. Last week I tried to upgrade to the last version v2.4.10 and unfortunately we still have issues with MySQL. With this version (v2.4.1) the icinga2 service starts, but the checks take forever and indeed in the icinga2.log file we saw the following pc-tdq-icinga-10 : version 2.4.10-1 [2016-06-01 16:19:03 +0200] information/IdoMysqlConnection: Query queue items: 987844, query rate: 0.7/s (42/min 46/5min 2251/15min); empty in infinite time, your database isn't able to keep up I installed also a virtual machine with an Icinga2 server chacking 314 hosts and 6338 and still we have the same issue with MySQL, so it is not related to the number of hosts and services being checked. Please let me know if you need further details or if you have any suggestion, we do need to use MySQL. The MySQL verson is Ver 14.14 Distrib 5.1.73. Thank you and best regards, Diana Production server =============== RAM 32 GB Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz [root@pc-tdq-icinga-20 ~]# icinga2 feature list Disabled features: debuglog gelf graphite icingastatus livestatus opentsdb perfdata syslog Enabled features: api checker command compatlog ido-mysql mainlog notification statusdata [root@pc-tdq-icinga-20 ~]# icinga2 -V icinga2 - The Icinga 2 network monitoring daemon (version: v2.3.11) Copyright (c) 2012-2015 Icinga Development Team (https://www.icinga.org) License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Application information: Installation root: /usr Sysconf directory: /etc Run directory: /var/run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /var/run/icinga2/icinga2.pid Application type: icinga/IcingaApplication System information: Operating system: Linux Operating system version: 2.6.32-573.8.1.el6.x86_64 Architecture: x86_64 Distribution: Scientific Linux CERN SLC release 6.7 (Carbon) Test server: ========== 32 GB RAM Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz [root@pc-tdq-icinga-10 ~]# icinga2 feature list Disabled features: debuglog gelf graphite icingastatus livestatus opentsdb perfdata syslog Enabled features: api checker command compatlog ido-mysql mainlog notification statusdata [root@pc-tdq-icinga-10 ~]# icinga2 -V icinga2 - The Icinga 2 network monitoring daemon (version: v2.4.10) Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/) License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Application information: Installation root: /usr Sysconf directory: /etc Run directory: /var/run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Modified attributes path: /var/lib/icinga2/modified-attributes.conf Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /var/run/icinga2/icinga2.pid System information: Platform: Scientific Linux CERN SLC Platform version: 6.7 (Carbon) Kernel: Linux Kernel version: 2.6.32-573.8.1.el6.x86_64 Architecture: x86_64 - Diana Scannicchio University of California, Irvine ATLAS TDAQ SysAdmin group Office: +41 22 76 75240 OnCall: 164851 _______________________________________________ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
