If it were me, I would forget all the printf and query stuff, and just make
the value of the alerting expression be the value you want.
expr: node_filesystem_avail_bytes and (.... some other trigger
expression ...)
This gives the value of the LHS, if there is a matching expression on the
RHS (exact same set of labels); it ignores the value on the RHS.
e.g.
expr: node_filesystem_avail_bytes and (node_filesystem_avail_bytes /
node_filesystem_size_bytes < 0.1)
Then I would use {{ $value | humanize1024 }} in the annotation, although
alternatively you could divide the expr to get MiB:
expr: (node_filesystem_avail_bytes/1024/1024) and
(node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1)
However if you want both the percentage *and* the raw MiB in the
annotation, then yes you have to jump through hoops. I've never used the
"query" filter, but if it's regular PromQL then you could do the arithmetic
there:
printf \"node_filesystem_avail_bytes{mountpoint='%s'}/1024/1024\"
.Labels.mountpoint | query | first | value
On Wednesday, 16 August 2023 at 08:31:13 UTC+1 Станислав Кузнецов wrote:
> Good afternoon!
> I have a problem.I want to display the remaining memory in alerts, which
> takes the value of the node_filesystem_avail_bytes metric and divides it by
> / 1024 / 1024 (to convert to Mb).
>
> Now my rule looks like this:
> - alert: HostOutOfDiskSpace_test
> expr: (node_filesystem_avail_bytes * 100) /
> node_filesystem_size_bytes > 0 and ON (instance, device, mountpoint)
> node_filesystem_readonly == 0 and {instance=~"10.22.22.18.*"}
> for: 1s
> labels:
> severity: high
> annotations:
> summary: "Host {{ $labels.instance }} out of memory \n - Device: {{
> $labels.device }} \n - Mountpoint - {{ $labels.mountpoint }}"
> description: "Node memory is filling up. Value = {{ $value | printf
> `%.2f` }} ({{ printf \"node_filesystem_avail_bytes{mountpoint='%s'}\"
> .Labels.mountpoint | query | first | value|humanize1024}})"
>
> But the value is displayed incorrectly. Prometheus shows the correct
> values of the node_filesystem_avail_bytes metrics, but in the notification
> it is not recalculated correctly. Here are examples of notifications from
> telegrams:
>
> 🔥 [PROBLEM] HostOutOfDiskSpace_test_1
> Severity: high
> Summary: Host 10.22.22.181:9100 out of memory
> - Device: /dev/sda1
> - Mountpoint - /boot
> Description: Node memory is filling up. Value = 72.62 (736.4Mi)
> Starts at: 2023-08-15 16:19:52.25 +0300 MSK
> ____________________
>
> 🔥 [PROBLEM] HostOutOfDiskSpace_test_1
> Severity: high
> Summary: Host 10.22.22.181:9100 out of memory
> - Device: tmpfs
> - Mountpoint - /run
> Description: Node memory is filling up. Value = 98.14 (298.2Mi)
> Starts at: 2023-08-15 16:19:52.25 +0300 MSK
> ____________________
>
> 🔥 [PROBLEM] HostOutOfDiskSpace_test_1
> Severity: high
> Summary: Host 10.22.22.181:9100 out of memory
> - Device: tmpfs
> - Mountpoint - /run/user/1000
> Description: Node memory is filling up. Value = 100.00 (298.8Mi)
> Starts at: 2023-08-15 16:19:52.25 +0300 MSK
> ____________________
>
> 🔥 [PROBLEM] HostOutOfDiskSpace_test_1
> Severity: high
> Summary: Host 10.22.22.181:9100 out of memory
> - Device: /dev/mapper/rhel-root
> - Mountpoint - /
> Description: Node memory is filling up. Value = 87.08 (31.48Gi)
> Starts at: 2023-08-15 16:19:52.25 +0300 MSK
> ____________________
>
> At the same time, Prometheus shows the following values:
> {device="/dev/mapper/rhel-root", fstype="xfs", instance="10.22.22.181:9100",
> mountpoint="/"} 15146.296875
> {device="/dev/sda1", fstype="xfs", instance="10.22.22.181:9100",
> mountpoint="/boot"} 736.39453125
> {device="tmpfs", fstype="tmpfs", instance="10.22.22.181:9100",
> mountpoint="/run"} 872.640625
> {device="tmpfs", fstype="tmpfs", instance="10.22.22.181:9100",
> mountpoint="/run/user/1000"} 177.83984375
>
> Here is the output from the OS:
> Filesystem Size Used Avail Use% Mounted on
> devtmpfs 871M 0 871M 0% /dev
> tmpfs 890M 0 890M 0% /dev/shm
> tmpfs 890M 17M 873M 2% /run
> tmpfs 890M 0 890M 0% /sys/fs/cgroup
> /dev/mapper/rhel-root 17G 2.2G 15G 13% /
> /dev/sda1 1014M 278M 737M 28% /boot
> tmpfs 178M 0 178M 0% /run/user/1000
>
> Reading the official documentation from the prometheus.io website, I
> realized that the problem was in the translation of the data. I need one
> rule to work for different devices and mount points. The construct {{
> printf \"node_filesystem_avail_bytes{mountpoint='%s'}\" .Labels.mountpoint
> | query | first | value | humanize1024 }} works the way I want it to, but
> the mountpoint string that contains the given value doesn't translate
> correctly to humanize1024. humanize Very far from real values, so I don't
> consider it.
>
> Maybe someone has come across this. How can I display
> node_filesystem_avail_bytes associated with a specific device and mount
> point, bypassing the humanize1024 function, but simply dividing by / 1024 /
> 1024 or some other conversion to MB or GB?
>
> Thank you for your responses!
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/c144fea2-ac7a-4b9c-b92b-6ae779ba16c7n%40googlegroups.com.
