hi,
i have a gke setup with monitoring setup. so i have the requirement of
using gpu nodes so i have configured the prometheus to get the gpu metrics.
but when i plot the query using the gpu metrics i got empty query result.
when i do the queries individually they are showing the data.
here are the result
i am using this gpu query ----->DCGM_FI_DEV_GPU_UTIL
when i combine with my deployment -----> sum by (namespace, pod,
label_seldon_deployment_id) (DCGM_FI_DEV_GPU_T) * on(namespace, pod)
group_left(label_seldon_deployment_id) sum by (namespace, pod,
label_seldon_deployment_id) (DCGM_FI_DEV_GPU_UTIL) * on(namespace, pod)
group_left(label_seldon_deployment_id)
kube_pod_labels{label_seldon_deployment_id="llm-testing-1"}
its not giving the result. showing empty result.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/fa4481bc-de7d-485f-938f-988909228fc0n%40googlegroups.com.
