I think you are confused between prometheus and pushgateway <https://github.com/prometheus/pushgateway>.
If you want to "push_to_gateway", then the thing you want to be pushing to is a running instance of *pushgateway* (which would be running on port 9999 in your example). Then separately, you run an instance of prometheus with a scrape job which scrapes the metrics from the pushgateway periodically. It will always scrape the most-recently-pushed value; this value is "sticky" if you don't push to it again. If the metric you want to store represents the status of the current or most-recently run ML job, like "started at time X" or "Y% through" or "successfully completed", then pushgateway is a reasonable approach. However if you're trying to push a *series* of timestamped values then pushgateway is not what you want. Therefore to make this work: - you need to run pushgateway on port 9999 - you need to run prometheus on a different port (9090 is normal) - you need to add a scrape job to your prometheus.yml to fetch metrics from the pushgateway The other way to "push" data directly into Prometheus is to use the remote_write protocol <https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations>, but that's a whole different ball game. In this case, your source needs to keep pushing data periodically (at least every 2 minutes), to stop the timeseries from becoming stale. On Friday, 9 June 2023 at 13:51:32 UTC+1 apoorve kalot wrote: > Hello, > > I currently intend to use prometheus python client library to push custom > metrics which are being generated through ML model inference. for now, in > the development environment, using following python code: (for now code > just push random values generated). > ``` > import random > import time > from prometheus_client import CollectorRegistry, Gauge, push_to_gateway > > class MetricsCollector: > def __init__(self, endpoint): > self.registry = CollectorRegistry() > self.gauges = {} > self.endpoint = endpoint > def add_metric(self, name): > gauge = Gauge(name, 'Description of the metric', registry=self.registry) > self.gauges[name] = gauge > def set_metric_value(self, name, value): > gauge = self.gauges.get(name) > if gauge is not None: > gauge.set(value) > def collect_metrics(self): > while True: > for name, gauge in self.gauges.items(): > # Generate random value using random library > value = random.random() > self.set_metric_value(name, value) > # Push metrics to the Prometheus endpoint > push_to_gateway(self.endpoint, job='my_job', registry=self.registry) > time.sleep(5) # Collect metrics every 5 seconds > > # Example usage > if __name__ == '__main__': > collector = MetricsCollector('http://0.0.0.0:9999') # Replace with your > Prometheus endpoint > # Add metrics > collector.add_metric('metric1') > collector.add_metric('metric2') > # Start collecting and pushing metrics > collector.collect_metrics() > ``` > And following default template for YML file: > ``` > # my global config > global: > scrape_interval: 15s # Set the scrape interval to every 15 seconds. > Default is every 1 minute. > evaluation_interval: 15s # Evaluate rules every 15 seconds. The default > is every 1 minute. > # scrape_timeout is set to the global default (10s). > > # Alertmanager configuration > alerting: > alertmanagers: > - static_configs: > - targets: > # - alertmanager:9093 > > # Load rules once and periodically evaluate them according to the global > 'evaluation_interval'. > rule_files: > # - "first_rules.yml" > # - "second_rules.yml" > > ``` > With initialising command as as : > ``` > ./prometheus --config.file="prometheus.yml" --web.listen-address=" > 0.0.0.0:9999" > ``` > > Then too it is throwing error: > ``` > File "/root/miniconda3/envs/mlops/lib/python3.7/urllib/request.py", line > 649, in http_error_default > raise HTTPError(req.full_url, code, msg, hdrs, fp) > urllib.error.HTTPError: HTTP Error 404: Not Found > ``` > > I tried sending api request to check whether the url is accessible with > following python code: > ``` > import requests > url = "http://0.0.0.0:9999"; > payload = {} > headers = {} > r = requests.request("GET", url, headers=headers, data=payload) > print(r) > ``` > > Which gives output: > ``` > <Response [200]> > ``` > which means that URL atleast exist. > > What can be possible problem associated with this though. > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a4f94e63-57af-46e7-926a-07de5bb03246n%40googlegroups.com.
