Hi Roberto,
thank you very much, this is the right parameter to toggle!
I have change it from "false" to "true" and now we have graphs working
again.
Have a good day
Francesco
Il 25/02/26 14:20, Roberto Valverde Cameselle ha scritto:
Hi Francesco,
Can you check if you have this feature set?
mgr advanced mgr/prometheus/exclude_perf_counters
Make sure that is set to true so then performance metrics are gathered
from the ceph-exporters and not from the manager.
If your cluster is small, you can also not add ceph-exporters and set
the above setting to false so the manager will be also exposing the
performance metrics.
Cheers
*-----------------------------------------------------------------*
*Roberto VALVERDE CAMESELLE*
IT Storage And Data Management
CERN, European Organization for Nuclear Research
Esplanade des Particules 1, Geneve (Switzerland)
------------------------------------------------------------------------
*From:* Francesco Usseglio Gaudi via ceph-users <[email protected]>
*Sent:* Tuesday, February 24, 2026 6:17 PM
*To:* [email protected] <[email protected]>
*Subject:* [ceph-users] metrics duplicate in prometheus after migrate
and upgrade to cephadm
Hi all,
we have a 10 nodes cluster used for S3 with radosgw.
Initially the cluster has been installed with ceph-ansible and was at
version 18.
Some months ago an external consultant converted to cephadm and upgrade
to 19.2.3
Almost everything is ok but we have some issues with dashboard and
grafana graphs.
In Dashboard → Cluster → OSDs → Overall Performance we can't see "OSD
Read Latencies" and "OSD Write Latencies" graphs, they have red triangle
icons with this error popup:
"execution: found duplicate series for the match group
{ceph_daemon=\"osd.0\"} on the right hand-side of the operation:
[{ceph_daemon=\"osd.0\",
cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\",
instance=\"ceph_cluster\", job=\"ceph\"}, {ceph_daemon=\"osd.0\",
cluster=\"6e06959e-3ef3-4017-a467-b1d482bc7269\", instance=\"node1\",
job=\"ceph-exporter\"}];many-to-many matching not allowed: matching
labels must be unique on one side"
After some my investigations I found out there are some duplicate
metrics in prometheus db, one coming from ceph (I think mgr module?) and
one from ceph-exporter, for example this metric:
ceph_osd_op_r_latency_sum
be there with two distinct values with different job labels (values are
exactly the same)
Every node has a running ceph-exporter container and there are 3 nodes
with mgr services.
I have tried to modify grafana dashboard queries to use only one of two
metrics specifying the job label but they are readonly (I can make a
copy and modifying it that works, but i don't know how to use this new
grafana dashboard in ceph dashboard).
I have read from ceph documentation and I understand ceph-exporter is no
longer required but on another ceph cluster recently set up with cephadm
version 19.2.3 from the very start, ceph-exporter container are presents
on every nodes, so I'm not sure if I should keep ceph-exporter
containers running
Which is the right way of extract these metrics from ceph?
Thank you all
Francesco Usseglio
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
