On 27/07/2023 15:51, Uvais Ibrahim wrote:
Hi Brain,
This is the query that I have used.
sum(scrape_samples_scraped)without(app,app_kubernetes_io_managed_by,clusterName,release,environment,instance,job,k8s_cluster,kubernetes_name,kubernetes_namespace,ou,app_kubernetes_io_component,app_kubernetes_io_name,app_kubernetes_io_version,kustomize_toolkit_fluxcd_io_name,kustomize_toolkit_fluxcd_io_namespace,application,name,role,app_kubernetes_io_instance,app_kubernetes_io_part_of,control_plane,beta_kubernetes_io_arch,beta_kubernetes_io_instance_type,
beta_kubernetes_io_os, failure_domain_beta_kubernetes_io_region,
failure_domain_beta_kubernetes_io_zone,kubernetes_io_arch,
kubernetes_io_hostname, kubernetes_io_os,
node_kubernetes_io_instance_type, nodegroup,
topology_kubernetes_io_region,
topology_kubernetes_io_zone,chart,heritage,revised,transit,component,namespace,
pod_name, pod_template_hash, security_istio_io_tlsMode,
service_istio_io_canonical_name,
service_istio_io_canonical_revision,k8s_app,kubernetes_io_cluster_service,kubernetes_io_name,route_reflector)
Which simply excluded every label but still I am getting a result like
this
{} 7525871918
I'm not sure what you are expecting, as that sounds about right. The
query is adding together all the different variants of the
scrape_samples_scraped metric (removing all the different labels), so if
that is indeed a list of every label the query is going to return a
value without any associated labels.
You want to be instead just graphing the raw scrape_samples_scraped
metric (no sum or without) and see how it varies over time. Is there a
particular job or target which has a huge increase in the graph, or new
series appearing? As to why that might happen it could be many different
reasons, but ideas could include:
* new version of software which increases number of exposed metrics (or
more granular labels)
* bug in software where a label is set to something with high
cardinality (e.g. there is a "path" label from a web app, which means a
potentially infinite cardinality, and you could have had a web scan
producing millions of combinations)
* lots of changes to the targets, such as new instances of software or
high churn of applications restarting
--
Stuart Clark
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/7423743a-8b29-625d-4472-6aa710cf1179%40Jahingo.com.
