Hi Michael, thanks for your quick reply!
>>> [...]
>>> 06/12/2015 15:35:08 :: wrote 441 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_ok.value
>>> in 0.00660 seconds
>>> 06/12/2015 15:35:08 :: wrote 442 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.min_execution_time.value
>>> in 0.00105 seconds
>>> 06/12/2015 15:35:08 :: wrote 442 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_hosts_flapping.value
>>> in 0.00077 seconds
>>> 06/12/2015 15:35:08 :: wrote 442 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_hosts_up.value
>>> in 0.00419 seconds
>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_unknown.value
>>> in 0.00075 seconds
>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.avg_latency.value
>>> in 0.00070 seconds
>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.max_execution_time.value
>>> in 0.00063 seconds
>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_pending.value
>>> in 0.00075 seconds
>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_warning.value
>>> in 0.00066 seconds
>>> [...]
>>
>> The extreme load drops to normal values immediately once I shut down either
>> of the satellites - it doesn't have to be the one that's doing the extreme
>> amount of perfdata writing.
>
> Does that pattern repeat?
Hundreds of time per second, as long as the second satellite is in operation.
And it's not only the 'icinga' check, (probably) all other checks are showing
the same behaviour as well ('disk', for instance):
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
> 1170210816 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
> 30408704 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
> 1170210816 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
> 30408704 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
> 1170210816 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
> 30408704 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
> 321149468672 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
> 1170210816 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
> 30408704 1449421237'.
> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
> 321149468672 1449421237'.
or 'load':
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
> 0.059999999999999998 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
> 0.080000000000000002 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
> 0.11 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
> 0.059999999999999998 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
> 0.080000000000000002 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
> 0.11 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
> 0.059999999999999998 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
> 0.080000000000000002 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
> 0.11 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
> 0.059999999999999998 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
> 0.080000000000000002 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
> 0.11 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
> 0.059999999999999998 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
> 0.080000000000000002 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
> 0.11 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
> 0.059999999999999998 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
> 0.080000000000000002 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
> 0.11 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
> 0.059999999999999998 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
> 0.080000000000000002 1449421231'.
> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
> 0.11 1449421231'.
And as I said, it immediately stops and drops down to normal behavour when I
stop one of the satellite endpoints.
> It seems your “icinga” check is running in a frequent interval producing that
> amount of performance data.
>
> Please show the service object for your “icinga” check configuration.
The services and templates are fairly standard:
template Service "generic-service" {
max_check_attempts = 5
check_interval = 1m
retry_interval = 30s
enable_perfdata = true
}
apply Service "icinga" {
import "generic-service"
check_command = "icinga"
command_endpoint = host.name
assign where "cluster-nodes" in host.groups
}
I definitely didn't do change about the check interval (yet). And it works
perfectly normal as long as there is only one satellite. It just starts going
bonkers as soon as there are two satellites.
> Object 'icinga2-satellite1.vm.hindenburgring.com!icinga' of type 'Service':
> % declared in '/etc/icinga2/zones.d/global/icinga2-services.conf', lines
> 12:1-12:22
> * __name = "icinga2-satellite1.vm.hindenburgring.com!icinga"
> * action_url = ""
> * check_command = "icinga"
> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
> lines 15:5-15:28
> * check_interval = 60
> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
> 12:5-12:23
> * check_period = ""
> * command_endpoint = "icinga2-satellite1.vm.hindenburgring.com"
> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
> lines 17:5-17:32
> * display_name = "icinga"
> * enable_active_checks = true
> * enable_event_handler = true
> * enable_flapping = false
> * enable_notifications = true
> * enable_passive_checks = true
> * enable_perfdata = true
> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
> 15:5-15:26
> * event_command = ""
> * flapping_threshold = 30
> * groups = [ ]
> * host_name = "icinga2-satellite1.vm.hindenburgring.com"
> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
> lines 12:1-12:22
> * icon_image = ""
> * icon_image_alt = ""
> * max_check_attempts = 5
> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
> 11:5-11:26
> * name = "icinga"
> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
> lines 12:1-12:22
> * notes = ""
> * notes_url = ""
> * package = "_etc"
> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
> lines 12:1-12:22
> * retry_interval = 30
> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
> 13:5-13:24
> * templates = [ "icinga", "generic-service" ]
> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
> lines 12:1-12:22
> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
> 10:1-10:34
> * type = "Service"
> * vars = null
> * volatile = false
> * zone = "satellite"
> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
> lines 12:1-12:22
Best regards,
Peter.
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
