Hi Michael (et al.), I don't want to be annoying, but does anyone have any other ideas how to further track this issue down? I could try to set up a minimised environment that still shows the problem, but I'd rather wait with that effort until I know that there isn't a bug somewhere in the code that's already known.
Cheers,
Peter.
> On 7 Dec 2015, at 11:19, Peter Eckel <li...@eckel-edv.de> wrote:
>
> Hi Michael,
>
> thanks for your quick reply!
>
>>>> [...]
>>>> 06/12/2015 15:35:08 :: wrote 441 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_ok.value
>>>> in 0.00660 seconds
>>>> 06/12/2015 15:35:08 :: wrote 442 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.min_execution_time.value
>>>> in 0.00105 seconds
>>>> 06/12/2015 15:35:08 :: wrote 442 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_hosts_flapping.value
>>>> in 0.00077 seconds
>>>> 06/12/2015 15:35:08 :: wrote 442 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_hosts_up.value
>>>> in 0.00419 seconds
>>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_unknown.value
>>>> in 0.00075 seconds
>>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.avg_latency.value
>>>> in 0.00070 seconds
>>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.max_execution_time.value
>>>> in 0.00063 seconds
>>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_pending.value
>>>> in 0.00075 seconds
>>>> 06/12/2015 15:35:08 :: wrote 443 datapoints for
>>>> icinga2.icinga2-satellite1_vm_hindenburgring_com.services.icinga.icinga.perfdata.num_services_warning.value
>>>> in 0.00066 seconds
>>>> [...]
>>>
>>> The extreme load drops to normal values immediately once I shut down either
>>> of the satellites - it doesn't have to be the one that's doing the extreme
>>> amount of perfdata writing.
>>
>> Does that pattern repeat?
>
> Hundreds of time per second, as long as the second satellite is in operation.
> And it's not only the 'icinga' check, (probably) all other checks are showing
> the same behaviour as well ('disk', for instance):
>
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
>> 1170210816 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
>> 30408704 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
>> 1170210816 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
>> 30408704 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
>> 1170210816 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
>> 30408704 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._vagrant.value
>> 321149468672 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._.value
>> 1170210816 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._boot.value
>> 30408704 1449421237'.
>> [2015-12-06 17:00:47 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.disk.disk.perfdata._mnt_hgfs.value
>> 321149468672 1449421237'.
>
> or 'load':
>
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
>> 0.059999999999999998 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
>> 0.080000000000000002 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
>> 0.11 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
>> 0.059999999999999998 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
>> 0.080000000000000002 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
>> 0.11 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
>> 0.059999999999999998 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
>> 0.080000000000000002 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
>> 0.11 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
>> 0.059999999999999998 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
>> 0.080000000000000002 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
>> 0.11 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
>> 0.059999999999999998 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
>> 0.080000000000000002 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
>> 0.11 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
>> 0.059999999999999998 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
>> 0.080000000000000002 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
>> 0.11 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load5.value
>> 0.059999999999999998 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load15.value
>> 0.080000000000000002 1449421231'.
>> [2015-12-06 17:00:46 +0000] debug/GraphiteWriter: Add to metric
>> list:'icinga2.icinga2-satellite1_vm_hindenburgring_com.services.load.load.perfdata.load1.value
>> 0.11 1449421231'.
>
> And as I said, it immediately stops and drops down to normal behavour when I
> stop one of the satellite endpoints.
>
>> It seems your “icinga” check is running in a frequent interval producing
>> that amount of performance data.
>>
>> Please show the service object for your “icinga” check configuration.
>
> The services and templates are fairly standard:
>
> template Service "generic-service" {
> max_check_attempts = 5
> check_interval = 1m
> retry_interval = 30s
>
> enable_perfdata = true
> }
>
> apply Service "icinga" {
> import "generic-service"
>
> check_command = "icinga"
>
> command_endpoint = host.name
>
> assign where "cluster-nodes" in host.groups
> }
>
> I definitely didn't do change about the check interval (yet). And it works
> perfectly normal as long as there is only one satellite. It just starts going
> bonkers as soon as there are two satellites.
>
>> Object 'icinga2-satellite1.vm.hindenburgring.com!icinga' of type 'Service':
>> % declared in '/etc/icinga2/zones.d/global/icinga2-services.conf', lines
>> 12:1-12:22
>> * __name = "icinga2-satellite1.vm.hindenburgring.com!icinga"
>> * action_url = ""
>> * check_command = "icinga"
>> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
>> lines 15:5-15:28
>> * check_interval = 60
>> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
>> 12:5-12:23
>> * check_period = ""
>> * command_endpoint = "icinga2-satellite1.vm.hindenburgring.com"
>> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
>> lines 17:5-17:32
>> * display_name = "icinga"
>> * enable_active_checks = true
>> * enable_event_handler = true
>> * enable_flapping = false
>> * enable_notifications = true
>> * enable_passive_checks = true
>> * enable_perfdata = true
>> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
>> 15:5-15:26
>> * event_command = ""
>> * flapping_threshold = 30
>> * groups = [ ]
>> * host_name = "icinga2-satellite1.vm.hindenburgring.com"
>> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
>> lines 12:1-12:22
>> * icon_image = ""
>> * icon_image_alt = ""
>> * max_check_attempts = 5
>> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
>> 11:5-11:26
>> * name = "icinga"
>> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
>> lines 12:1-12:22
>> * notes = ""
>> * notes_url = ""
>> * package = "_etc"
>> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
>> lines 12:1-12:22
>> * retry_interval = 30
>> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
>> 13:5-13:24
>> * templates = [ "icinga", "generic-service" ]
>> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
>> lines 12:1-12:22
>> % = modified in '/etc/icinga2/zones.d/global/templates.conf', lines
>> 10:1-10:34
>> * type = "Service"
>> * vars = null
>> * volatile = false
>> * zone = "satellite"
>> % = modified in '/etc/icinga2/zones.d/global/icinga2-services.conf',
>> lines 12:1-12:22
>
>
> Best regards,
>
> Peter.
> _______________________________________________
> icinga-users mailing list
> icinga-users@lists.icinga.org
> https://lists.icinga.org/mailman/listinfo/icinga-users
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
