If you can, deploy (a) Prometheus into the cluster itself. The easiest way to manage that is using the Prometheus operator, but if that is not possible, you can configure it directly using relabeling, as in this example[0].
This Prometheus can scrape the various targets. You have a few options from there: You can use this directly, reaching it through the load balancer, or through a Grafana deployed to the same cluster. Or use remote write to push to another Prometheus or other metric store. In this case you can run the in-cluster Prometheus in the pared down agent mode. This also works if you run e.g. one Prometheus per namespace. On the more complex but full featured end, you can use Thanos to tie multiple servers in multiple clusters together with a long term store. What is appropriate for you depends on the size of your setup, what you want monitor, and the restrictions that your admins impose. I hope this gives you some pointers to discuss with them! Best, Matthias [0]: https://github.com/prometheus/prometheus/blob/main/documentation/examples/prometheus-kubernetes.yml On Fri, Apr 15, 2022, 17:19 [email protected] <[email protected]> wrote: > Hello, > > I'd like to use Prometheus to monitor my (wrapped) k8s services, but > unfortunately the admin won't let me connect directly to individual pods, I > can just access them via load balancing. > > In this case, I guess I have to use the Pushgateway. It seems to be > working, however timeseries for old pods persist (the "instance" label is a > random string). I am thinking of creating a program to periodically delete > them from the Pushgateway. > > This is not an ideal situation but I guess this is the best I can do. Or > am I missing something? > > Thanks, > > -- > Adriano > > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/e009b2af-53e3-4c43-8d3c-64b3cc29041bn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/e009b2af-53e3-4c43-8d3c-64b3cc29041bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAMV%3D_gbxxQCcF_k0qoagRwTxP4KJLd3Kd8fEVjvXLitgEpSwJA%40mail.gmail.com.
