Ok, I discovered the root cause of the issue. The features-enabled directory configurations were not being read by Icinga. Checking the directory I see:
# ls -la /etc/icinga2/features-enabled total 8 drwxr-x---. 2 icinga icinga 4096 Sep 9 17:54 . drwxr-x---. 11 icinga icinga 4096 Sep 10 02:13 .. lrwxrwxrwx. 1 root root 30 Sep 9 14:49 api.conf -> ../features-available/api.conf lrwxrwxrwx. 1 root root 34 Sep 9 14:43 checker.conf -> ../features-available/checker.conf lrwxrwxrwx. 1 root root 34 Sep 9 14:49 command.conf -> ../features-available/command.conf lrwxrwxrwx. 1 root root 35 Sep 9 17:54 debuglog.conf -> ../features-available/debuglog. conf lrwxrwxrwx. 1 root root 36 Sep 9 14:43 ido-mysql.conf -> ../features-available/ido-mysql.conf lrwxrwxrwx. 1 root root 34 Sep 9 14:43 mainlog.conf -> ../features-available/mainlog.conf lrwxrwxrwx. 1 root root 39 Sep 9 14:43 notification.conf -> ../features-available/notification.conf Oddly, features-available appears to have the correct permissions: # ls -la /etc/icinga2/features-available/ total 72 drw-r-----. 2 icinga icinga 4096 Sep 10 02:13 . drwxr-x---. 11 icinga icinga 4096 Sep 10 02:13 .. -rw-r--r--. 1 icinga icinga 312 Sep 9 20:42 api.conf -rw-r-----. 1 icinga icinga 131 Sep 5 14:32 checker.conf -rw-r-----. 1 icinga icinga 158 Sep 5 14:32 command.conf -rw-r-----. 1 icinga icinga 177 Sep 5 14:32 compatlog.conf -rw-r-----. 1 icinga icinga 255 Sep 5 14:32 debuglog.conf -rw-r-----. 1 icinga icinga 227 Sep 5 14:32 gelf.conf -rw-r-----. 1 icinga icinga 210 Sep 5 14:32 graphite.conf -rw-r-----. 1 icinga icinga 223 Sep 5 14:32 icingastatus.conf -rw-r-----. 1 icinga icinga 313 Sep 9 20:42 ido-mysql.conf -rw-r-----. 1 icinga icinga 144 Sep 5 14:32 livestatus.conf -rw-r-----. 1 icinga icinga 179 Sep 5 14:32 mainlog.conf -rw-r-----. 1 icinga icinga 151 Sep 5 14:32 notification.conf -rw-r-----. 1 icinga icinga 210 Sep 5 14:32 opentsdb.conf -rw-r-----. 1 icinga icinga 166 Sep 5 14:32 perfdata.conf -rw-r-----. 1 icinga icinga 243 Sep 5 14:32 statusdata.conf -rw-r-----. 1 icinga icinga 127 Sep 5 14:32 syslog.conf Removing the symlink and manually copying the files over from features-available fixed the issue. If anyone has a reason why the original symlink didn't work, I'd be greatly interested. Now for a second problem--> both instances are now running, but the second instance essentially freezes up once connected to the cluster, i.e. SSH sessions become unstable and unusuable. Any ideas on where to start troubleshooting? From: icinga-users [mailto:icinga-users-boun...@lists.icinga.org] On Behalf Of Dan Thagard (3P) Sent: Wednesday, September 9, 2015 5:27 PM To: icinga-users@lists.icinga.org Subject: [icinga-users] Icinga2 ApiListener Not Running, No IDO Data Written I'm trying to configure a single-zone, master-master HA setup with no sync of configuration files (using Chef to replicate the config data between instances, so no config master) and a separate mysql database. I was able to successfully deploy a single Icinga2 instance, but I've been struggling with setting up HA. At this point, no checks are running, and no data is being written to the IDO. I can access the IcingaWeb2 interface, but the site is contains no check information (I can login and browse). Running icinga2 troubleshoot, I get the following: [root@ip-10-0-1-198 init.d]# icinga2 troubleshoot critical/cli: Cannot access path ''. critical/troubleshoot: Failed to collect enabled and/or disabled features. Check /etc/icinga2/features-available /etc/icinga2/features-enabled warning/troubleshoot: No loggers found, check whether you enabled any logging features warning/troubleshoot: zones.conf not found. If you are using a zones.conf somewhere but the default path please provide it with your support request Started collection at 2015-09-09 20:55:18 Finished collection at 2015-09-09 20:55:19 Took 1.7313578128814697 seconds General log file: '/var/log/icinga2/troubleshooting-2015-09-09_20:55:18.log' Please compress the files before uploading them,, for example: # tar czf troubleshoot.tar.gz /var/log/icinga2/troubleshooting-2015-09-09_20:55:18.log* The relevant parts of the log are here: ============== GENERAL INFORMATION ============== Application version: v2.3.10 Installation root: /usr Sysconf directory: /etc Run directory: /var/run Local state directory: /var Package data directory: /usr/share/icinga2 State path: /var/lib/icinga2/icinga2.state Objects path: /var/cache/icinga2/icinga2.debug Vars path: /var/cache/icinga2/icinga2.vars PID path: /var/run/icinga2/icinga2.pid Application type: icinga/IcingaApplication ============== LOGS AND CRASH REPORTS ============== ######################## No loggers found, check whether you enabled any logging features ######################## No crash logs found in /var/log/icinga2/crash/ An output of relevant files/dirs: #ls -la /etc/icinga2/features-enabled/ total 8 drwxr-x---. 2 icinga icinga 4096 Sep 9 17:54 . drwxr-x---. 11 icinga icinga 4096 Sep 9 20:54 .. lrwxrwxrwx. 1 root root 30 Sep 9 14:12 api.conf -> ../features-available/api.conf lrwxrwxrwx. 1 root root 34 Sep 9 14:11 checker.conf -> ../features-available/checker.conf lrwxrwxrwx. 1 root root 34 Sep 9 14:12 command.conf -> ../features-available/command.conf lrwxrwxrwx. 1 root root 35 Sep 9 17:54 debuglog.conf -> ../features-available/debuglog.conf lrwxrwxrwx. 1 root root 36 Sep 9 14:11 ido-mysql.conf -> ../features-available/ido-mysql.conf lrwxrwxrwx. 1 root root 34 Sep 9 14:11 mainlog.conf -> ../features-available/mainlog.conf lrwxrwxrwx. 1 root root 39 Sep 9 14:11 notification.conf -> ../features-available/notification.conf # ls -la /etc/icinga2/pki/ total 24 drwxr-x---. 2 icinga icinga 4096 Sep 9 20:33 . drwxr-x---. 11 icinga icinga 4096 Sep 9 20:54 .. -rw-r--r--. 1 icinga icinga 1696 Sep 9 20:33 ca.crt -rw-r--r--. 1 icinga icinga 1692 Sep 9 20:33 i-2e2b55c7.crt -rw-r--r--. 1 icinga icinga 1586 Sep 9 20:33 i-2e2b55c7.csr -rw-------. 1 icinga icinga 3243 Sep 9 20:33 i-2e2b55c7.key # ls -la /var/log/icinga2/ total 264 drwxr-x---. 4 icinga icingacmd 4096 Sep 9 21:08 . drwxr-xr-x. 10 root root 4096 Sep 9 14:12 .. drwxr-x---. 3 icinga icingacmd 4096 Sep 9 14:11 compat drwxr-x---. 2 icinga icinga 4096 Sep 5 14:32 crash -rw-------. 1 icinga icingacmd 0 Sep 9 14:12 error.log -rw-r--r--. 1 root root 172032 Sep 9 20:55 startup.log -rw-r--r--. 1 icinga icinga 81452 Sep 9 20:55 troubleshooting-2015-09-09_20:55:18.log #cat /etc/icinga2/features-available/ido-mysql.conf /** * The db_ido_mysql library implements IDO functionality * for MySQL. */ library "db_ido_mysql" object IdoMysqlConnection "ido-mysql" { user = "<user_redacted>" password = "<password_redacted>" host = "<host_redacted>" database = "icinga_ido" } enable_ha = "true" # cat /var/log/icinga2/startup.log [2015-09-09 20:55:06 +0000] information/cli: Icinga application loader (version: v2.3.10) [2015-09-09 20:55:06 +0000] information/cli: Loading application type: icinga/IcingaApplication [2015-09-09 20:55:06 +0000] information/Utility: Loading library 'libicinga.so' [2015-09-09 20:55:06 +0000] information/ConfigCompiler: Compiling config file: /etc/icinga2/icinga2.conf [2015-09-09 20:55:06 +0000] information/ConfigCompiler: Compiling config file: /etc/icinga2/constants.conf [2015-09-09 20:55:06 +0000] information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/itl [2015-09-09 20:55:06 +0000] information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/command.conf [2015-09-09 20:55:06 +0000] information/Utility: Loading library 'libmethods.so' [2015-09-09 20:55:06 +0000] information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/command-icinga.conf [2015-09-09 20:55:06 +0000] information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/timeperiod.conf ....... [2015-09-09 20:55:06 +0000] information/ConfigItem: Committing config items [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere! [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/conf.d/notifications.conf: 11:1-11:45) for type 'Notification' does not match anywhere! [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule 'mail-icingaadmin' (in /etc/icinga2/conf.d/notifications.conf: 19:1-19:48) for type 'Notification' does not match anywhere! [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule 'backup-downtime' (in /etc/icinga2/conf.d/downtimes.conf: 5:1-5:52) for type 'ScheduledDowntime' does not match anywhere! [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule 'ping6' (in /etc/icinga2/conf.d/services.conf: 40:1-40:21) for type 'Service' does not match anywhere! [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule 'ssh' (in /etc/icinga2/conf.d/services.conf: 53:1-53:19) for type 'Service' does not match anywhere! [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 64:1-64:65) for type 'Service' does not match anywhere! [2015-09-09 20:55:07 +0000] warning/ApplyRule: Apply rule '' (in /etc/icinga2/conf.d/services.conf: 72:1-72:53) for type 'Service' does not match anywhere! [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 56 CheckCommand(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 6 NotificationCommand(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 3 TimePeriod(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 217 HostGroup(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 3 ServiceGroup(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 1146 Host(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 1618 Service(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 3 User(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 3514 Notification(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 3 UserGroup(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 2 Endpoint(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 1 Zone(s). [2015-09-09 20:55:07 +0000] information/ConfigItem: Checked 1 IcingaApplication(s). [2015-09-09 20:55:07 +0000] information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars' [2015-09-09 20:55:07 +0000] information/DynamicObject: Restoring program state from file '/var/lib/icinga2/icinga2.state' [2015-09-09 20:55:07 +0000] information/DynamicObject: Restored 6573 objects. Loaded 0 new objects without state. [2015-09-09 20:55:07 +0000] information/ConfigItem: Triggering Start signal for config items # iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination icinga all -- anywhere anywhere http all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination Chain http (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:http /* http */ ACCEPT tcp -- anywhere anywhere tcp dpt:https /* http */ Chain icinga (1 references) target prot opt source destination ACCEPT tcp -- anywhere anywhere tcp dpt:5665 /* icinga */ ACCEPT udp -- anywhere anywhere udp dpt:5665 /* icinga */ # telnet <host_redacted> 3306 Trying <host_redacted>... Connected to <host_redacted>. Escape character is '^]'. A "netstat -tulpen | grep icinga" returns nothing. Also, querying the database for instances returns nothing: select * from icinga_instances LIMIT 0, 1000 0 row(s) returned Any ideas or help would be greatly appreciated! Please let me know if I need to provide any further information. Thanks, Dan
_______________________________________________ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
