2 years have passed, did you manage to resolve this Chalapathi?
I'm curious because I seem to have the exact same problem, with the
interaction between the istio proxy metrics and a Prometheus operator
installation. I'm seeing the metrics when curl'ing the stats/prometheus
endpoints, but they are not collected by prometheus.
On Monday, August 16, 2021 at 8:19:22 AM UTC+2 Venkata Bhagavatula wrote:
> Hi All,
>
> I am seeing some time series from the target are getting dropped and not
> written to tsdb. Initially i thought it might be related to wrong metric
> relabel configs, but i have removed all the relabel configs, even then the
> time series is not being written.
> Target is giving the time series.
>
> Following the is curl output from the target:
> istio_requests_total{response_code="200",reporter="destination",
> *source_workload="ind-cpro"*,source_workload_namespace="csd-ns",source_principal="unknown",source_app="cpro",source_version="unknown",destination_workload="ind-nrfp",destination_workload_namespace="csd-ns",destination_principal="unknown",destination_app="nrfp",destination_version="unknown",destination_service="csd-nnrfp.csd-ns.svc.cluster.local",destination_service_name="csd-nnrfp",destination_service_namespace="csd-ns",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="none",source_canonical_service="cpro",destination_canonical_service="nrfp",source_canonical_revision="2.15.1",destination_canonical_revision="latest"}
>
> 3017
> istio_requests_total{response_code="200",reporter="source",
> *source_workload="ind-nrfp"*,source_workload_namespace="csd-ns",source_principal="unknown",source_app="nrfp",source_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",destination_principal="unknown",destination_app="unknown",destination_version="unknown",destination_service="
>
> abc.default.svc.cluster.local",destination_service_name="abc.default.svc.cluster.local",destination_service_namespace="unknown",request_protocol="grpc",response_flags="-",grpc_response_status="0",connection_security_policy="unknown",source_canonical_service="nrfp",destination_canonical_service="unknown",source_canonical_revision="latest",destination_canonical_revision="latest"}
>
> 276
>
> Prometheus job configuration is as follows(ip address is masked):
> - honor_labels: true
> job_name: chal
> metrics_path: /stats/prometheus
> static_configs:
> - targets:
> - abc.xyz.def.pqr:15090
>
> In the above time series from target, the time series with
> *source_workload="ind-nrfp"
> *is not present in prometheus. There are no warnings/errors seen in the
> debug logs
>
> Can you let me know how to debug this issue?
>
> Thanks n Regards,
> Chalapathi.
>
>
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/8c0d8f44-93f8-45bf-a2aa-a078982f7d59n%40googlegroups.com.
