[icinga-users] Dependency design advice for a lot of clusters service?
Hello Guys, I'm using quite a big icinga2 configuration (about 3000 services), without any dependencies. This results in a huge mess of notifications when something goes wrong. I try to look for best practices with Icinga2 Dependencies, but all I find on the web is basic understanding with a single example. What I need is deeper information, best practices on how to manage such a config file. How do I design an dependency ? How do I manage my dependencies ( especially for cluster service ) ? for example : I have ten Apache Cassandra server. Each server also has a script to check the cluster service. If seven number of servers in the cluster criticcal alarm will genarated . However, this alarm comes from the 10 servers. The same problem exists in redis cluster service. I have a six redis server. Likewise, there is a script that controls the cluster service. ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
Re: [icinga-users] Metric graphing with Icinga 2
Thanks for the pointers. The debug log shows this for the missing metrics: [2016-08-11 17:47:48 -0500] warning/GraphiteWriter: Ignoring invalid perfdata value: in=0.00B/s;1024.00;1024.00;0.00 [2016-08-11 17:47:48 -0500] warning/GraphiteWriter: Ignoring invalid perfdata value: out=965.00B/s;2048.00;2048.00;0.00 [2016-08-11 17:47:48 -0500] warning/GraphiteWriter: Ignoring invalid perfdata value: in=0.00B/s;1024.00;1024.00;0.00 [2016-08-11 17:47:48 -0500] warning/GraphiteWriter: Ignoring invalid perfdata value: out=195.00B/s;2048.00;2048.00;0.00 Why would these entries be invalid? Because the in/out metrics are appended with "B/s"? Thanks, Michael On 08/12/2016 01:49 AM, Michael Friedrich wrote: Hi, On 12 Aug 2016, at 00:56, Michael Martin wrote: I have just now implemented graphite/grafana at our installation. The graphs look great. One question, though. After running for a little while to get the feel of it, we turned on another metric, specifically, we are now logging switch traffic with check_mrtgtraf. But, the performance metrics aren't making it into graphite. We turned on debuglog (icinga2 feature enable debuglog) and can see the perf data being properly returned, but it just ain't making it into graphite. Poking around in graphite's web app, nothing. Not there. Grafana can't see it, so we can't graph it. Any idea what I might need to do to get this going? I've restarted everything, done "graphite-manage syncdb", etc. What am I missing? * Check the debug log for “GraphiteWriter” sending the metrics * Verify that carbon-cache actually processes the data. By default the metrics/minute are set to a pretty low default value (max_updates_per_second, max_creates_per_minute, see my tuning over at the vagrant boxes - https://github.com/Icinga/icinga-vagrant/commit/6a6c53552c74040479331566dd683e3782ec8966) Kind regards, Michael Thanks, Michael On 08/10/2016 01:20 PM, Joseph L. Casale wrote: You should look into grafana as graphite frontend. Or when graphite doesn't scale anymore grafana and prometheus. Grafana certainly looks impressive and polished. I imagine Prometheus overlaps with Icinga a bit. What are peoples thoughts on influxdb? Thanks, jlc ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users -- Michael Friedrich, DI (FH) Senior Developer NETWAYS GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB18461 http://www.netways.de | michael.friedr...@netways.de ** OSBConf 2016 - September - osbconf.org ** ** OSMC 2016 - November - netways.de/osmc ** ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
Re: [icinga-users] Using check_multi with Icinga2
On Thu, Aug 11, 2016 at 11:53 PM, Michael Friedrich
wrote:
> That as such won’t work since the custom attributes values are computed at
> config compile time. You are looking for runtime calculated values - which is
> why you need to turn the custom attribute values into a function.
>
> Something like that:
>
> vars.cluster_label = {{ get_host("vagrant-minion").state }}
> vars.cluster_data = {{ get_host("vagrant-minion").state }}
>
Ah, got it! This works. Thanks so much for the clarification.
Now I'm running into a new challenge. I would like to construct an
array of hostnames and then reference this array inside a service
object, as follows:
vars.elb["x"] = [ "xx.ap-northeast-1.compute.amazonaws.com" ,
"xy.ap-northeast-1.compute.amazonaws.com" ]
apply Service "clusterx" {
if (!vars.elb["x"] { log("Array not defined") }
...
}
The log statement shows me that this array is not defined in the
service definition. What am I doing wrong?
___
icinga-users mailing list
icinga-users@lists.icinga.org
https://lists.icinga.org/mailman/listinfo/icinga-users
Re: [icinga-users] Metric graphing with Icinga 2
Yes, the "B/s" string in the performance data was tripping up Graphite. I removed that, and all is working as expected now. Thanks for your help. On 08/12/2016 01:49 AM, Michael Friedrich wrote: Hi, On 12 Aug 2016, at 00:56, Michael Martin wrote: I have just now implemented graphite/grafana at our installation. The graphs look great. One question, though. After running for a little while to get the feel of it, we turned on another metric, specifically, we are now logging switch traffic with check_mrtgtraf. But, the performance metrics aren't making it into graphite. We turned on debuglog (icinga2 feature enable debuglog) and can see the perf data being properly returned, but it just ain't making it into graphite. Poking around in graphite's web app, nothing. Not there. Grafana can't see it, so we can't graph it. Any idea what I might need to do to get this going? I've restarted everything, done "graphite-manage syncdb", etc. What am I missing? * Check the debug log for “GraphiteWriter” sending the metrics * Verify that carbon-cache actually processes the data. By default the metrics/minute are set to a pretty low default value (max_updates_per_second, max_creates_per_minute, see my tuning over at the vagrant boxes - https://github.com/Icinga/icinga-vagrant/commit/6a6c53552c74040479331566dd683e3782ec8966) Kind regards, Michael Thanks, Michael On 08/10/2016 01:20 PM, Joseph L. Casale wrote: You should look into grafana as graphite frontend. Or when graphite doesn't scale anymore grafana and prometheus. Grafana certainly looks impressive and polished. I imagine Prometheus overlaps with Icinga a bit. What are peoples thoughts on influxdb? Thanks, jlc ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users -- Michael Friedrich, DI (FH) Senior Developer NETWAYS GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg Tel: +49 911 92885-0 | Fax: +49 911 92885-77 CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB18461 http://www.netways.de | michael.friedr...@netways.de ** OSBConf 2016 - September - osbconf.org ** ** OSMC 2016 - November - netways.de/osmc ** ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users ___ icinga-users mailing list icinga-users@lists.icinga.org https://lists.icinga.org/mailman/listinfo/icinga-users
Re: [icinga-users] Using check_multi with Icinga2
You’re going to have to move the attribute declaration _into_ the apply rule:
apply Service "clusterx" {
vars.elb["x"] = [ "xx.ap-northeast-1.compute.amazonaws.com" ,
"xy.ap-northeast-1.compute.amazonaws.com" ]
if (!vars.elb["x"] { log("Array not defined") }
...
}
Otherwise your assignment would create a global variable „vars“ – which is not
associated with any service and therefore has no effect.
This seems like a good opportunity to mention
https://www.icinga.org/2015/11/12/icinga-2-script-debugger/ :)
-Original Message-
From: icinga-users on behalf of Michael
Martinez
Reply-To: "icinga-users@lists.icinga.org"
Date: Friday 12 August 2016 at 20:10
To: "icinga-users@lists.icinga.org"
Subject: Re: [icinga-users] Using check_multi with Icinga2
On Thu, Aug 11, 2016 at 11:53 PM, Michael Friedrich
wrote:
> That as such won’t work since the custom attributes values are computed
at config compile time. You are looking for runtime calculated values - which
is why you need to turn the custom attribute values into a function.
>
> Something like that:
>
> vars.cluster_label = {{ get_host("vagrant-minion").state }}
> vars.cluster_data = {{ get_host("vagrant-minion").state }}
>
Ah, got it! This works. Thanks so much for the clarification.
Now I'm running into a new challenge. I would like to construct an
array of hostnames and then reference this array inside a service
object, as follows:
vars.elb["x"] = [ "xx.ap-northeast-1.compute.amazonaws.com" ,
"xy.ap-northeast-1.compute.amazonaws.com" ]
apply Service "clusterx" {
if (!vars.elb["x"] { log("Array not defined") }
...
}
The log statement shows me that this array is not defined in the
service definition. What am I doing wrong?
___
icinga-users mailing list
icinga-users@lists.icinga.org
https://lists.icinga.org/mailman/listinfo/icinga-users
--
Gunnar Beutner
Senior Developer
NETWAYS GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg
Tel: +49 911 92885-0 | Fax: +49 911 92885-77
CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB18461
http://www.netways.de | gunnar.beut...@netways.de
** OSBConf 2016 - September - osbconf.org **
** OSMC 2016 - November - netways.de/osmc **
___
icinga-users mailing list
icinga-users@lists.icinga.org
https://lists.icinga.org/mailman/listinfo/icinga-users
Re: [icinga-users] Using check_multi with Icinga2
On Fri, Aug 12, 2016 at 12:32 PM, Gunnar Beutner
wrote:
> You’re going to have to move the attribute declaration _into_ the apply rule:
Hmmm..
Let me explain what I need to do, and maybe the folks here can help
suggest how to achieve it;
Outside of Icinga, I gather a list of amazon instances and the ELBs
that they are attached to. I am using Jinja templates to produce the
Icinga host objects from this list. In order to associate each ELB to
which hosts belong to it, I have done two things:
a) in each host object, have a vars._elb variable that contains the
names of the ELBs the host belongs to
b) outside of the host objects, created a "global" elb["elbname"]
array that contains the hostnames of the hosts that belong to it
Now, I need to create a Service object (for each ELB) that contains a
vars.cluster_data variable which contains the hosts in the ELB, as
follows:
vars.cluster_data = {{ get_host("hostname1").state + "," +
get_host("hostname2").state + "," +
}}
I was trying to do this by referencing the "global" elb array as I
mentioned, but apparently this global data is not available within the
Service definition.
What woudl be the right approach?
Perhaps instead of creating an ELB array, I should create a hostgroup
for each ELB, and put the ELB's members as members of the hostgroup.
Then I can reference them in my Service definition?
--
---
Michael Martinez
http://www.michael--martinez.com
___
icinga-users mailing list
icinga-users@lists.icinga.org
https://lists.icinga.org/mailman/listinfo/icinga-users
Re: [icinga-users] Using check_multi with Icinga2
How about something like this?:
object Host "ec2-instance1" {
import "generic-host"
address = "192.168.2.36"
vars.elbs = [ "elb1", "elb2" ]
}
object Host "ec2-instance2" {
import "generic-host"
address = "192.168.2.36"
vars.elbs = [ "elb1", "elb2" ]
}
var ELBs = [
"elb1",
"elb2"
]
for (elb in ELBs) {
object Host elb {
import "generic-host"
}
object Service "cluster" use(elb) {
import "generic-service"
host_name = elb
check_command = "dummy"
vars.ec2_instances = []
for (host in get_objects(Host)) {
if (elb in host.vars.elbs) {
vars.ec2_instances.add(host.name)
}
}
vars.cluster_data = {{
var states = []
for (instance in macro("$ec2_instances$")) {
states.add(get_host(instance).state)
}
return states.join(",")
}}
vars.cluster_state = {{
var worst_state = 0
for (instance in macro("$ec2_instances$")) {
var state = get_host(instance).state
if (state > worst_state) {
worst_state = state
}
}
return worst_state
}}
vars.cluster_info = {{
var worst_state = macro("$cluster_state$")
var instances = []
for (instance in macro("$ec2_instances$")) {
if (get_host(instance).state == worst_state) {
instances.add(instance)
}
}
var states = [ "OK", "Warning", "Critical", "Unknown" ]
return "Instances " + instances.join(", ") + " are in state " +
states[worst_state]
}}
vars.dummy_state = "$cluster_state$"
vars.dummy_text = "$cluster_info$"
}
}
This automatically figures out which instances belongs to each ELB by iterating
over all hosts and checking their ‚elbs’ custom variable. I’ve also added a few
additional custom variables (cluster_state, cluster_info) which demonstrate how
you can use use check_dummy to generate your own check results – without using
another external plugin.
-Original Message-
From: icinga-users on behalf of Michael
Martinez
Reply-To: "icinga-users@lists.icinga.org"
Date: Friday 12 August 2016 at 23:15
To: "icinga-users@lists.icinga.org"
Subject: Re: [icinga-users] Using check_multi with Icinga2
On Fri, Aug 12, 2016 at 12:32 PM, Gunnar Beutner
wrote:
> You’re going to have to move the attribute declaration _into_ the apply
rule:
Hmmm..
Let me explain what I need to do, and maybe the folks here can help
suggest how to achieve it;
Outside of Icinga, I gather a list of amazon instances and the ELBs
that they are attached to. I am using Jinja templates to produce the
Icinga host objects from this list. In order to associate each ELB to
which hosts belong to it, I have done two things:
a) in each host object, have a vars._elb variable that contains the
names of the ELBs the host belongs to
b) outside of the host objects, created a "global" elb["elbname"]
array that contains the hostnames of the hosts that belong to it
Now, I need to create a Service object (for each ELB) that contains a
vars.cluster_data variable which contains the hosts in the ELB, as
follows:
vars.cluster_data = {{ get_host("hostname1").state + "," +
get_host("hostname2").state + "," +
}}
I was trying to do this by referencing the "global" elb array as I
mentioned, but apparently this global data is not available within the
Service definition.
What woudl be the right approach?
Perhaps instead of creating an ELB array, I should create a hostgroup
for each ELB, and put the ELB's members as members of the hostgroup.
Then I can reference them in my Service definition?
--
---
Michael Martinez
http://www.michael--martinez.com
___
icinga-users mailing list
icinga-users@lists.icinga.org
https://lists.icinga.org/mailman/listinfo/icinga-users
--
Gunnar Beutner
Senior Developer
NETWAYS GmbH | Deutschherrnstr. 15-19 | D-90429 Nuernberg
Tel: +49 911 92885-0 | Fax: +49 911 92885-77
CEO: Julian Hein, Bernd Erk | AG Nuernberg HRB18461
http://www.netways.de | gunnar.beut...@netways.de
** OSBConf 2016 - September - osbconf.org **
** OSMC 2016 - November - netways.de/osmc **
___
icinga-users mailing list
icinga-users@lists.icinga.org
https://lists.icinga.org/mailman/listinfo/icinga-users
