Re: [prometheus-users] Re: Unknown auth 'public_v1'

Hmm, maybe we should add an empty config check (no modules, no auths) to the startup. On Fri, Jun 27, 2025 at 9:20 AM 'Brian Candler' via Prometheus Users < prometheus-users@googlegroups.com> wrote: > You're not passing --config.file to snmp_exporter. > > Therefore it will only attempt to pick up

Re: [prometheus-users] Drop metrics based on value?

No, there is not. There is a proposal for at-scrape-time rule evaluations which may help with this. But unless it were tens to hundreds of thousands of series I just ignore these kinds of things in the TSDB. It's just not worth my time to worry about a few megabytes of memory to store the data.

Re: [prometheus-users] Ritesh_patel_SRE

This is not an appropriate list for resume / job postings. On Fri, Mar 28, 2025 at 9:05 AM ritesh patel wrote: > Hello All, > > My self Ritesh patel, currently I am looking for a new job. I have total > 10 years of experience in SRE. I worked with couple of monitoring tools > like Grafana,Promet

Re: [prometheus-users] How to handle SNMP TruthValue type?

- regex: '(.*)' > value: '$1' > > On Thu, Mar 27, 2025 at 10:30 AM Ben Kochie wrote: > >> Metrics are always a mess. ^.^ >> >> It's a constant toil to improve metrics on every system. >> >> But, yea, SNMP te

Re: [prometheus-users] How to handle SNMP TruthValue type?

ng on snake_case instead of CamelCase. I would have to include > the original name in the help text so you can reference the MIB in case of > any questions in the future. But this takes a lot of time to customize > each module... > > On Thu, Mar 27, 2025 at 3:19 AM Ben Kochie wrot

Re: [prometheus-users] How to handle SNMP TruthValue type?

t; MIB? > > On Thursday, 27 March 2025 at 09:24:56 UTC Ben Kochie wrote: > >> I added some numeric features recently for this specific case. >> >> Here's an example: >> >> https://github.com/prometheus/snmp_exporter/blob/15269c2a5788db885178a8f98de28e7ce

Re: [prometheus-users] How to handle SNMP TruthValue type?

I added some numeric features recently for this specific case. Here's an example: https://github.com/prometheus/snmp_exporter/blob/15269c2a5788db885178a8f98de28e7ce10ed0a9/generator/generator.yml#L848-L851 On Thu, Mar 27, 2025 at 1:20 AM 'Elliott Balsley' via Prometheus Users < prometheus-users@g

Re: [prometheus-users] Re: Why does SNMP exporter show uppercase MAC?

The only place where I can think of would be in the textual convention RFCs. * https://www.rfc-editor.org/rfc/rfc1213.html * https://www.rfc-editor.org/rfc/rfc2579.html A quick scan doesn't seem to indicate a case preference. On Wed, Mar 26, 2025 at 9:58 AM 'Brian Candler' via Prometheus Users <

Re: [prometheus-users] snmp_exporter and non unique indexes

This looks like a bug in MIB. Ask the vendor to make sure the MIB matches the data returned by the device. On Mon, Mar 17, 2025 at 11:11 AM Hisham Attar wrote: > I have this Hitachi HCP device that reports status of ssds per node, SNMP > responds at a cluster level and contains data for each no

Re: [prometheus-users] Re: Spreading single alertmanager cluster nodes over multiple geographical regions

n Monday, March 3, 2025 at 7:03:30 AM UTC-5 Ben Kochie wrote: > >> Part of the Prometheus/Alertmanager design is to better survive WAN >> split-brain. >> >> IMO, running a wide Alertmanager cluster is a good idea when you have a >> wide network. The AM gossip proto

Re: [prometheus-users] Re: Repeated Scrape Values From Prometheus With Range Selection

> So don't want the repeated duplicate data Why? What problem are you trying to solve? On Thu, Mar 6, 2025 at 9:28 AM Suvendu Nayak wrote: > Thanks Brain, I can do this on client side with Oracle SQLs but the > problem is when Prometheus gets those data it scrapes in every 15 secs and > most of

Re: [prometheus-users] Re: Spreading single alertmanager cluster nodes over multiple geographical regions

Part of the Prometheus/Alertmanager design is to better survive WAN split-brain. IMO, running a wide Alertmanager cluster is a good idea when you have a wide network. The AM gossip protocol and deduplication is designed to fail open in the event of a split brain. The only thing you have to be awa

Re: [prometheus-users] Verifying CPU Usage % and Memory Usage % of all pods from prometheus data source on Grafana with cluster

container~="" means you're ignoring all of the containers in the pod. Did you mean to use container!=""? Also make sure to read this article about CPU limits . On Sat, Feb 8, 2025 at 8:09 PM Akash Rai wrote: > Hi Team, > I am new to Promethe

Re: [prometheus-users] Consistent backup of TSDB data

The accepted best practice is to automate the creation of VMs via infra as code and configuration management. Then only backup the necessary data. This has a large number of advantages. * You don't backup standard system files / binaries, reducing the cost of your backup storage space. * You don't

Re: [prometheus-users] Consistent backup of TSDB data

That doesn't really follow best practice for backups, but in order to get a consistent dataset you should call the snapshot API. https://prometheus.io/docs/prometheus/latest/querying/api/#tsdb-admin-apis On Sat, Feb 1, 2025 at 3:46 AM Doug Meredith wrote: > I'm running Prometheus on a VM and sy

Re: [prometheus-users] Promethues v2.x and Prometheus v3.x Tracks

incompatible. > > And till when is the support for v2.55 ? > There is none, it is now an old release. > > On Monday, 20 January 2025 at 15:36:34 UTC+5:30 Ben Kochie wrote: > >> On Mon, Jan 20, 2025 at 11:01 AM Harpreet Singh >> wrote: >> >>> Hello Co

Re: [prometheus-users] Promethues v2.x and Prometheus v3.x Tracks

On Mon, Jan 20, 2025 at 11:01 AM Harpreet Singh wrote: > Hello Community > > Since Promethues v3 has been released and currently I am using v2.53. And > to upgrade to v3, I first have to update to v2.55 to make the TSDB ready > for v3. It is not strictly required to upgrade to v2.55.x. It is on

Re: [prometheus-users] High Memory Usage with Prometheus WAL Despite --enable-feature=memory-snapshot-on-shutdown

tpoint","value":52087},{"name":"__name__","value":44903},{"name":"name","value":33040},{"na > > me":"container_id","value":20944},{"name":"replicaset","value":15

Re: [prometheus-users] High Memory Usage with Prometheus WAL Despite --enable-feature=memory-snapshot-on-shutdown

I would recommend getting a heap snapshot and posting it to https://pprof.me . curl -o heap.pprof http://localhost:9090/debug/pprof/heap Also including http://localhost:9090/tsdb-status would help. On Thu, Nov 7, 2024 at 9:53 AM Bhanu Prakash wrote: > Hello Prometheus Community, > > I’m encoun

Re: [prometheus-users] Help generating a complete snmp.yml file for FortiGate with snmp_exporter

Maybe easier, use the fortigate exporter. https://github.com/bluecmd/fortigate_exporter On Tue, Nov 5, 2024 at 5:30 PM Miguel Ángel Barrientos Ramírez < miguel.barrien...@liddev.com> wrote: > I'm trying to generate an *snmp.yml* file for *snmp_exporter* that can > pull all possible metrics from

Re: [prometheus-users] SNMP EXPORTER GENERATOR ERRORS

Did you read the error messages? Your MIBDIRS are missing a number of MIBs in order to satisfy all the requirements. Either find the missing MIBs, or set the `MIBDIRS` env var to point at the generator example "mibs" dir that is created with `make mibs`. On Mon, Sep 9, 2024 at 1:50 PM Mitchell L

Re: [prometheus-users] PromQL redirection

This is somewhat how Thanos works. It has a gRPC federating protocol that allows you to pull in metric data from multiple TSDBs and glue the results together in the Thanos Query engine. The Thanos Sidecar already supports Prometheus remote read protocol, so it could be a shim for what you need.

Re: [prometheus-users] Re: I am having an error while checking the alert manager status

Try https://prometheus-community.github.io/ansible/branch/main/ On Fri, Aug 30, 2024 at 11:19 AM chinelo Ufondu wrote: > I just had to uninstall alert manager, its stressing me out > Please i need a good guide on installing alertmanager, i want to start > afresh > The guide i have seen so far is

Re: [prometheus-users] Oddity with v0.xxx tags

That tagging scheme makes me sad. On Tue, Aug 20, 2024, 19:19 Bjoern Rabenstein wrote: > On 20.08.24 06:35, Ben Kochie wrote: > > Yes, this is a workaround for Go major versioning changes. > > > > See: https://github.com/prometheus/prometheus/pull/12900 > > A

Re: [prometheus-users] Oddity with v0.xxx tags

Yes, this is a workaround for Go major versioning changes. See: https://github.com/prometheus/prometheus/pull/12900 On Mon, Aug 19, 2024 at 8:52 PM 'Brian Candler' via Prometheus Users < prometheus-users@googlegroups.com> wrote: > I have just noticed a load of tags in the prometheus repo for v0.

Re: [prometheus-users] Graph Issues grafana and peometheus

ric in question, >> if I use probe_success, should I write it as >> avg_over_time(probe_duration_seconds{instance="xx", >> job="blackbox_exporter"}[30m])? >> Cloud you recommend me about function prometheus ? >> >> Thanks in advance, >> >>

Re: [prometheus-users] Re: node exporter's data collection frequency

Best practice if you don't want to include NFS mounts is to exclude it from the list of filesystem types. --collector.filesystem.fs-types-exclude="^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue| *nfs* |nsfs|overlay|proc|procfs|pstore|rpc_pipefs|

Re: [prometheus-users] Re: SNMP Exporter - Gathering MAC and IP per port

SNMP has the concept of a "Context Name" that is part of the walk, in addition to the community and other security parameters. This can be included in the auth section of the config[0], or as a URL parameter in the latest release[1]. [0]: https://github.com/prometheus/snmp_exporter/tree/main/gene

Re: [prometheus-users] Counter or Gauge metric?

This is one of those tricky situations where there's not a strict correct answer. For power-on-hours I would probably go with a gauge. * You don't really have a "perfect" monotonic counter here. * I would also include the serial number label as well, just for uniqueness identification sake. * Powe

Re: [prometheus-users] Graph Issues grafana and peometheus

This is Prometheus staleness handling. In order to allow queries to function, Prometheus graph query evaluation (range query) is actually a series of query evaluations over time. Each one is independent of the next. In order to support millisecond accurate timestamps, Prometheus will look back up

Re: [prometheus-users] Prometheus as deployment

We use the VerticalPodAutoscaler to auto-scale Prometheus instances. This would probably be more reliable than trying to deal with horizontal sharding. We also have a controller that shards by namespace, which helps isolate tenants on the cluster from causing problems with each other. On Thu, Jul

Re: [prometheus-users] A potential issue with your Blackbox DNS probes

Thanks for the detailed post. Sounds like a feature request/bug report. I would file an issue on GitHub, this should be easily solved. https://github.com/prometheus/blackbox_exporter/issues On Wed, Jun 26, 2024 at 12:19 AM Chris Siebenmann < cks.prom-users...@cs.toronto.edu> wrote: > To make a l

Re: [prometheus-users] Uptime SLA in percentage for metric

A PromQL query like "mymetric == bool 2" will return 1 when the value is > 2, and 0 otherwise. > > You'll likely need to run this inside a subquery if you're doing time > range aggregation over it. But if Grafana is doing the summarization that > might not be neces

Re: [prometheus-users] Uptime SLA in percentage for metric

IMO you need to fix your service metrics. Prometheus best practice is to follow the pattern of probe_success. Boolean values are far easier to handle. On Mon, Jun 24, 2024 at 2:36 PM Raúl Lopez wrote: > Hello, > I need to know in percentage the time my service has been available in the > last mo

Re: [prometheus-users] SNMP Exporter - MessPC Ethernetbox - Configuration

The thing you want to do is use the snmp_exporter generator. This will correctly parse the MIB and produce a valid snmp_exporter module config. Something like this works: modules: ethernetbox: walk: - version - location - sensorTable lookups: - source_indexes: [sensorinde

Re: [prometheus-users] SNMP exporter

You need to find out from your vendor what OIDs/MIBs are useful for the "GM200 device". On Wed, Jun 12, 2024 at 6:53 AM BHARATH KUMAR wrote: > Hello All, > > I have two servers A and B. server A is a GM200 device where SNMP agent is > enabled and nothing is installed there. > > And in server B I

Re: [prometheus-users] Scalability Limits of Single Prometheus Instance

To start, NFS is not supported. Only local disk storage. Q1: Prometheus today can scale to about 100M series, but operates a bit better below 50M series Q2: Infinite, the 2.0 TSDB has no practical storage limit. Q3: I've heard about instances with upwards of 50k targets. Q4: Query performance c

Re: [prometheus-users] Prometheus Metrics storage

Prometheus uses an inverted index to map names to series. I highly recommend watching this talk: https://promcon.io/2017-munich/talks/storing-16-bytes-at-scale/ On Mon, Jun 10, 2024 at 3:39 PM akshay sharma wrote: > Hi All, > > I'm working on Prometheus performance optimization. And simultaneou

Re: [prometheus-users] Reseting Histogram

There is no need to reset in order to get current average latencies. That's the beauty of cumulative counters. The PromQL would look like this: rate(my_histogram_seconds_sum[5m]) / rate(my_histogram_seconds_count[5m]) You can adjust whatever range vector value to want in order to control the ave

Re: [prometheus-users] how to get count of no.of instance

You can use the `up` metric sum(...) / count(up{job="kafka"}) On Fri, May 24, 2024 at 5:53 PM Sameer Modak wrote: > Hello Team, > > I want to know the no of instance data sending to prometheus. How do i > formulate the query . > > > Basically i have below working query but issues is we have 6

Re: [prometheus-users] OTel/Prometheus Interoperability Survey

pped, just always use base unit" > kind of answer here? > > Kind Regards, > Bartek Plotka (@bwplotka) > > On Thursday, May 2, 2024 at 7:26:54 PM UTC+1 Ben Kochie wrote: > >> Hmm, there's one question that's hard to answer because it doesn't follow &

Re: [prometheus-users] Does anyone have any examples of what a postgres_exporter.yml file is supposed to look like?

The postgres_exporter queries configuration is deprecated, you don't need it to monitor postgres servers anymore. On Tue, May 7, 2024 at 10:24 PM Christian Sanchez wrote: > Hello, all. > > I've started to learn Prometheus and found out about the > postgres_exporter. I'd like to include metrics f

Re: [prometheus-users] snmp generator.yml fails error err="cannot find oid '1.22610.2.4.1.2.2' to walk

used the cmd below to get the generator executable file: >>>>>>>> >>>>>>>> make generate >>>>>>>> >>>>>>>> then to run the generator ran this cmd: >>>>>>>> &

Re: [prometheus-users] OTel/Prometheus Interoperability Survey

Hmm, there's one question that's hard to answer because it doesn't follow normal conventions: Options are: node_time node_time_nanoseconds There is no option for the real Prometheus metric: node_time_seconds :( On Thu, May 2, 2024 at 5:02 PM Bartłomiej Płotka wrote: > Hi! > > The OpenTelemetry

Re: [prometheus-users] snmp generator.yml fails error err="cannot find oid '1.22610.2.4.1.2.2' to walk

e" module=a10 >> ts=2024-04-27T17:34:03.115Z caller=main.go:134 level=error msg="Error >> generating config netsnmp" err="cannot find oid >> 'axSysSecondaryVersionOnDisk' to walk >> ``` >> >> Also if I try generating the if-mib objects from if-mib module th

Re: [prometheus-users] Prometheus Alert on Switch Port Saturation

lay of my bandwidth values, for reference i am polling Meraki Switches > via SNMP > > Im going to give the above alert a try and see if that works as expected > > On Friday, April 19, 2024 at 10:05:32 AM UTC-4 Ben Kochie wrote: > >> Two things, >> >> Full-duplex

Re: [prometheus-users] Prometheus Alert on Switch Port Saturation

Two things, Full-duplex means you shouldn't add up `ifHCOutOctets` and `ifHCInOctets`. What you probably want is two alerts, one for "Port In Saturation" and one for "Port Out Saturation". Because ifHighSpeed is megabits/sec, you'll have to do a bit more math. If you want to deal with half-duplex

Re: [prometheus-users] Re: Need urgent help!!! Want to modify tags "keys" to lowercase scraping from Cloudwatch-Exporter in Prometheus before sending to Mimir #13912

Prometheus can lower/upper in relabeling. https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config On Thu, Apr 18, 2024 at 10:28 AM 'Brian Candler' via Prometheus Users < prometheus-users@googlegroups.com> wrote: > > Need urgent help!!! > > See https://www.catb.or

Re: [prometheus-users] Generator snmp_exporter return error 500 in prometheus

te-strg-data-1.net.globecast.com/>SNMPv2-MIB::sysLocation.0 >>> = STRING: STRGSNMPv2-MIB::sysServices.0 = INTEGER: >>> 6SNMPv2-MIB::sysORLastChange.0 = Timeticks: (0) >>> 0:00:00.00SNMPv2-MIB::sysORID.1 = OID: >>> SNMPv2-SMI::enterprises.9.7.129SNMPv2-MIB::sysORID.2 = OID: >>> SNMPv2-SMI::enterprises.9.7.11

Re: [prometheus-users] Generator snmp_exporter return error 500 in prometheus

If you use `snmp_exporter --log.level=debug`, what do the logs say? On Mon, Apr 15, 2024 at 10:38 PM Nicolas wrote: > Hello, > I have a strange error and I hope you can help me, or maybe there is a > problem with the snmp_exporter generator in the latest version. > > I'm using snmp_exporter vers

Re: [prometheus-users] Issues with Prometheus AlertManager Configuration

This is not a support team, this is a users discussion list. If you require professional support, please see our support page: https://prometheus.io/support-training/ On Sat, Apr 6, 2024 at 7:07 PM Baburao Ganpatrao < baburaoganpatrao...@gmail.com> wrote: > Dear Support Team, > > I am currently

Re: [prometheus-users] query for time series misses samples (that should be there), but not when offset is used

On Thu, Apr 4, 2024 at 7:03 PM Christoph Anton Mitterer wrote: > Hey Chris, Brian. > > Thanks for your replies/confirmations. > > > On Sunday, March 24, 2024 at 8:16:14 AM UTC+1 Ben Kochie wrote: > > Yup, this is correct. Prometheus sets the timestamp of the sample at t

Re: [prometheus-users] Alertmanager Config File For Cluster HA

No, the Prometheus design is that anything in the config file needs to be dynamic / runtime (SIGHUP) changeable. Things that are only possible to set at startup are handled by flags. This allows for easy differentiation between static and dynamic options. The listen addresses are currently only se

Re: [prometheus-users] How to isolate the data per "customer/tenant"

There are a couple of options. https://github.com/prometheus-community/prom-label-proxy https://thanos.io/tip/operating/multi-tenancy.md/ On Thu, Mar 28, 2024 at 5:01 PM Jorge Visentini wrote: > Hi guys! > > I'm noob in Prometheus so I am studying to deploy in my environment. > I have a particu

Re: [prometheus-users] [PushProx] Current container image?

Oh, I just realized, that is also the wrong location. The correct location for pushprox is https://hub.docker.com/r/prometheuscommunity/pushprox On Thu, Mar 28, 2024 at 11:30 AM Ben Kochie wrote: > There hasn't been a release since 2020. There are not enough maintainers > of the proj

Re: [prometheus-users] [PushProx] Current container image?

There hasn't been a release since 2020. There are not enough maintainers of the project. On Wed, Mar 27, 2024 at 1:16 AM Daz Wilkin wrote: > Folks -- > > PushProx has recent commits > > (19-Jun-202

Re: [prometheus-users] [snmp-exporter] when will --config.expand-environment-variables be available?

This feature is not required for PCI-DSS. There are many users of snmp_exporter who have PCI-DSS compliance and do not need it. On Tue, Mar 19, 2024 at 4:55 PM ohey...@gmail.com wrote: > Hi, > > > looking for this feature "--config.expand-environment-variables" to get > PCI-DSS compliance config

Re: [prometheus-users] query for time series misses samples (that should be there), but not when offset is used

Yup, this is correct. Prometheus sets the timestamp of the sample at the start of the scrape. But since it's an ACID compliant database, the data is not queryable until after it's been fully ingested. This is intentional, because the idea is that whatever atomicity is desired by the target is hand

Re: [prometheus-users] Re: better way to get notified about (true) single scrape failures?

I usually recommend throwing out any "But this is how Icinga does it". thinking. The way we do things in Prometheus for this kind of thing is to simply think about "availability". For any scrape failures: avg_over_time(up[5m]) < 1 For more than one scrape failure (assuming 15s intervals)

Re: [prometheus-users] snmp_exporter: mibs, indexes, and tables, oh my

> lgpEnvControlTemperature OBJECT-IDENTITY > STATUS current > DESCRIPTION > "Temperature measurement used as control feedback." > ::= { lgpEnvTemperatureWellKnown 1 } > > *lgpEnvReturnAirTemperature*OBJECT-

Re: [prometheus-users] Correlation between snmp scrape time and massive rate output for ifHCInOctets

dentical". Again, this is very much up to your remote storage implementation to figure out. > > Though I would love to contribute I’m not fluent enough in Go to offer any > meaningful assistance :). > > Thanks > Nick > > On Sat, 16 Mar 2024 at 09:38, Ben Kochie

Re: [prometheus-users] Correlation between snmp scrape time and massive rate output for ifHCInOctets

gt;> >> Alexander Wilke schrieb am Samstag, 16. März 2024 um 09:08:44 UTC+1: >> >>> Check File Format example. >>> >>> Time Out, retries, max-repetition. >>> >>> I use Repetition 50 or 100 with Cisco, retries 0 and Time Out 1s or >>> 500ms

Re: [prometheus-users] snmp_exporter: mibs, indexes, and tables, oh my

This is a pretty standard lookup. But the MIB entry for lgpEnvTemperatureDescrDegC is a bit strange. You can override the type easily to make it a DisplayString. But, based on your output, I'm guessing the device doesn't return a useful description. Can you provide an SNMP walk comparison? Here's

Re: [prometheus-users] blackbox_exporter 0.24.0 and smokeping_prober 0.7.1 - DNS cache "nscd" not working

All of the Prometheus components you're talking about are statically compiled Go binaries. These use Go's native DNS resolution. It does not use glibc. So maybe looking for solutions related to Golang and nscd would help. I've not looked into this myself. But on the subject of node local DNS cache

Re: [prometheus-users] Import/Export asynchronously

If you can make outbound TCP connections, you can use remote write. On Tue, Mar 12, 2024 at 5:46 PM T F wrote: > Hello, > > I would like to monitor applications runing on several private client > network. > The only way to communicate with them is to use sftp. > We can run promotheus agent on cl

[prometheus-users] Re: drop all some metrics based on regex

lutely correct but I don't have access to a lot of the servers > so I am trying to drop them on the prometheus side > > On Monday, March 11, 2024 at 1:39:18 PM UTC-7 Ben Kochie wrote: > >> relabel actions are exclusive. Drop means keep everything but X. Keep >> means dr

[prometheus-users] Re: drop all some metrics based on regex

relabel actions are exclusive. Drop means keep everything but X. Keep means drop everything but X. For your exact problem, there is already a node_exporter flag to handle this. ./node_exporter --collector.systemd.unit-include="(ssh|apache)" This will also be more efficient because it it will o

Re: [prometheus-users] Smokeping_prober CPU usage optimization possible?

GOGC is still empty. > > 1.) Is this environment variable set correctly? > 2.) Is the variable still working? > 3.) If it is still working can I apply it only to smokeping_prober but not > other services like prometheus? Sounds like higher GOGC has tradeoffs for > queries in the pro

Re: [prometheus-users] Smokeping_prober CPU usage optimization possible?

e main CPU consumers are snmp_exporter and smokeping. > > Ben Kochie schrieb am Sonntag, 25. Februar 2024 um 19:22:35 UTC+1: > >> Looking at the CPU profile, I'm seeing almost all the time spent in the >> Go runtime. Mostly the ICMP packet receiving code and garbage collection.

Re: [prometheus-users] Smokeping_prober CPU usage optimization possible?

. >> >> Smokeping_Prober Heap: >> >> >> https://pprof.me/a1e7400d32859dbc217e2182398485df/?profileType=profile%3Aalloc_objects%3Acount%3Aspace%3Abytes&dashboard_items=icicle >> >> >> >> smokeping_prober profile30s >> >> >> https:

Re: [prometheus-users] Optimal solution for storing 3 years of data from 300 hosts in prometheus server

val is 20s. So if i query data for last 3 years, can i assume > that this might take ~ 40MiB of memory per series ? and these operators > will not have any significant memory consumption? > > > > Regards > Puneet > > On Wednesday 21 February 2024 at 12:18:30 UTC+5:30 Ben K

Re: [prometheus-users] Optimal solution for storing 3 years of data from 300 hosts in prometheus server

at, and I don't understand why you're dividing it by CPU seconds. For things that you want to graph over long periods of time, you can use recording rules to generate pre-computed data that is easier to query. https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/ &

Re: [prometheus-users] Optimal solution for storing 3 years of data from 300 hosts in prometheus server

Prometheus needs a minimum of about 4KiB per "active series". Retention policy doesn't affect the memory usage very much. If you have 10,000 metrics per server and 300 servers, that's 3 million series. 3 million * 4KiB = 11.4GiB of memory. Of course, you will also need some page cache and such.

Re: [prometheus-users] Prometheus MultiTenancy / separation of metrics / separation of passwords

Take a look at https://github.com/prometheus-community/prom-label-proxy You can create separated datasources that connect to the label proxy instead of Prometheus directly. On Tue, Feb 20, 2024 at 6:04 AM Alexander Wilke wrote: > Hello, > > in our company I maintain a Prometheus Server to Monit

Re: [prometheus-users] Smokeping_prober CPU usage optimization possible?

Best thing you can do is capture some pprof data. That will show you what it's spending the time on. :9374/debug/pprof/heap :9374/debug/pprof/profile?seconds=30 You can post the results to https://pprof.me/ for sharing. On Tue, Feb 20, 2024 at 6:22 AM Alexander Wilke wrote: > Hello, > I am run

Re: [prometheus-users] blackbox_exporter - how to simplify my configuration

This is where a configuration management software like Ansible comes in. You can write the configuration with a for loop that expands the lists. https://galaxy.ansible.com/ui/repo/published/prometheus/prometheus/ On Sun, Feb 18, 2024 at 1:08 AM Alexander Wilke wrote: > Hello, > > In our DataCen

Re: [prometheus-users] snmp_exporter 0.25.0 + and prometheus 2.49.1 with "%" in label value - format issue

> - it feels like it started with the monitoring of f5 devices via SNMP. > No, this is just a failed string conversion. So you get the default hex conversion instead. I don't know what your error is, but I am fairly sure this is unrelated to Prometheus or SNMP data. > Ben Kochie

Re: [prometheus-users] Prometheus alert evaluation, are they instant queries?

; then reduce it to an instant vector, e.g. > > expr: avg_over_time(snmp_scrape_duration_seconds[5m]) >= 3 > > On Saturday 3 February 2024 at 16:04:56 UTC Ben Kochie wrote: > >> All rule evaluations are instant queries. You do all the "reducer >> functions" in

Re: [prometheus-users] Prometheus alert evaluation, are they instant queries?

All rule evaluations are instant queries. You do all the "reducer functions" in PromQL itself. For example, you can use subquery syntax to do something like `avg_over_time()`. On Sat, Feb 3, 2024 at 5:02 PM 'Andrew Dedesko' via Prometheus Users < prometheus-users@googlegroups.com> wrote: > Hi, >

Re: [prometheus-users] Re: snmp_exporter-0.20 cannot monitor SNMP V3?

; > I dont think it is correct as I'm still getting the following errors > > msg="Error parsing config file" err="yaml: unmarshal errors:\n line 2: > field fortigate not found in type config.Config\n line 6410: field idrac > not found in type config.Config" >

Re: [prometheus-users] Re: snmp_exporter-0.20 cannot monitor SNMP V3?

Did you read the migration doc? https://github.com/prometheus/snmp_exporter/blob/main/auth-split-migration.md On Fri, Jan 19, 2024 at 5:10 PM Nicholas Smith < nicholas.john.smit...@gmail.com> wrote: > Actually struggling with this myself since the change to auth split > migration in v0.23.0 > >

Re: [prometheus-users] Is there a reason why Prometheus chunk file size getting increased overtime?

Also please include more information. * Prometheus version. * All command line flags. On Thu, Jan 18, 2024 at 11:02 AM Ben Kochie wrote: > Can you graph these two metrics, over the time range you are talking about? > > prometheus_tsdb_head_series

Re: [prometheus-users] Is there a reason why Prometheus chunk file size getting increased overtime?

Can you graph these two metrics, over the time range you are talking about? prometheus_tsdb_head_series rate(prometheus_tsdb_head_samples_appended_total[1h]) On Thu, Jan 18, 2024 at 10:55 AM akbill wrote: > Hi all, > > Our customer has been using the Prometheus for years. The configurations >

Re: [prometheus-users] Smokeping_prober 0.7.1 - amount or buckets

More buckets cost more to store and process, but thankfully there is now Prometheus "native histograms", which give you high resolution for less cost. https://prometheus.io/docs/prometheus/latest/feature_flags/#native-histograms https://www.usenix.org/conference/srecon23emea/presentation/rabenste

Re: [prometheus-users] Why is snmp_exporter not recognizing the custom OID added?

You need to look at your generator output. I'm guessing there were errors. The HUAWEI-ENTITY-EXTENT-MIB OIDs you listed are in a table, so you can't *get* them, you need to *walk* them. When you generate, the output should have indexes that need to be used. You need to make sure your vendor MIBs

Re: [prometheus-users] Maximum targets for exporter

evices this is enterprise > network and servers/VMs with more CPUs are no problem. > > Ben Kochie schrieb am Freitag, 12. Januar 2024 um 21:50:57 UTC+1: > >> Those sound like reasonable amounts for those exporters. >> >> I've heard of people hitting thousands of SNMP

Re: [prometheus-users] Maximum targets for exporter

Those sound like reasonable amounts for those exporters. I've heard of people hitting thousands of SNMP devices from the snmp_exporter. Since the exporters are in Go, they scale well. But if it's not enough, the advantage of their design means they can be deployed horizontally. You could run seve

Re: [prometheus-users] Prometheus exporters in Grafana Agent

The Grafana agent isn't really meant for complex/large setups. If you have a large/dynamic target list you're probably better off just running normal Prometheus and exporters. On Thu, Jan 11, 2024 at 10:09 AM 'Elliott Balsley' via Prometheus Users < prometheus-users@googlegroups.com> wrote: > I n

Re: [prometheus-users] Re: Support for --tsdb.too-far-in-future.time-window in Prometheus

I think the problem here is that they have the system clock set incorrectly, intentionally. It's not tracking UTC with a local timezone set. But it's tracking a local timezone and the system thinks that is UTC. So Prometheus thinks UTC is some random local timezone, not real UTC. For the record,

Re: [prometheus-users] Inconsistency in handling IPv6 literals

Oh, I forgot to add, it would be helpful to file issues on the affected repos. On Fri, Dec 15, 2023 at 8:03 AM Ben Kochie wrote: > Yea, that's some inconsistency we should fix. I agree, we should allow > square brackets everywhere, but only require them if a port is required. >

Re: [prometheus-users] Inconsistency in handling IPv6 literals

Yea, that's some inconsistency we should fix. I agree, we should allow square brackets everywhere, but only require them if a port is required. Consistently using https://pkg.go.dev/net#SplitHostPort should help. On Thu, Dec 14, 2023 at 7:01 PM 'Brian Candler' via Prometheus Users < prometheus-us

Re: [prometheus-users] Help with Sum of hourly average

Without knowing the units, it's hard to answer what exactly you're trying to get. The value is an instantaneous watts and you want watt-hours? avg_over_time(snmp_value{metric="Consomation"}[24h]) * 24 This will give you the watt-hours over the 24 hours. On Tue, Dec 5, 2023 at 11:48 AM Malek Ba

Re: [prometheus-users] Re: Prometheus memory increases after retarting k8s target

It is. You can clearly see in your graphs that the Go memstats goes back down to the prior level. Go is a garbage collected language, memory use is going to fluctuate over time as Prometheus operates and GC happens. It's not an exact value and never will be. Memory use also depends on queries tha

Re: [prometheus-users] probe_success VS up

One more thing to talk about is that the Prometheus ecosystem assumes and follows the "Fail Fast" principle[0]. Best practice[1] in Prometheus is to fail the whole scrape and return a 5xx error if any part of the data collection fails. For simple exporters this is typical. The reason for this is t

Re: [prometheus-users] probe_success VS up

Fantastic summary. This would actually make a really nice addition to the "guides" section of the Prometheus docs. https://github.com/prometheus/docs/tree/main/content/docs/guides On Tue, Nov 28, 2023 at 11:18 AM 'Brian Candler' via Prometheus Users < prometheus-users@googlegroups.com> wrote: >

Re: [prometheus-users] APC UPS Error: "was collected before with the same name and label values"

ything. That might be why that one is working. So now the > question on the newer models is how to get it so they can be working? > Remove that OID? > > root@server# snmpwalk -v1 -c public > 1.3.6.1.4.1.318.1.1.1.2.3.10.2.1.8 > root@server:# > On Monday, November 27, 2023 at 10:34:3

Re: [prometheus-users] APC UPS Error: "was collected before with the same name and label values"

1.1.1.2.3.10.2.1.8.9.1.2 = STRING: "01/01/2000" > SNMPv2-SMI::enterprises.318.1.1.1.2.3.10.2.1.8.10.1.1 = STRING: > "01/01/2000" > SNMPv2-SMI::enterprises.318.1.1.1.2.3.10.2.1.8.10.1.2 = STRING: > "01/01/2000" > SNMPv2-SMI::enterprises.318.1.1.1.2.3.10.2

Re: [prometheus-users] Re: Federation of aggregated metrics

I'd say less neglected, but more obsoleted by. Remote Write and Thanos Sidecar are the more functional modern replacements for the original Federation method. On Wed, Nov 22, 2023 at 1:01 PM Bryan Boreham wrote: > Federation is a bit of a neglected feature. The Thanos project is rather > more p

Re: [prometheus-users] APC UPS Error: "was collected before with the same name and label values"

Can you find upsHighPrecBatteryPackCartridgeReplaceDate in your snmp.yml? What version of PowerNet-MIB do you have? This is what I get when I generate your config: - name: upsHighPrecBatteryPackCartridgeReplaceDate oid: 1.3.6.1.4.1.318.1.1.1.2.3.10.2.1.8 type: DisplayString h

Re: [prometheus-users] SNMP lookup not working

Would it be better to attempt this recursively, so if it fails to find the > name in the last branch, then it looks one branch up, and so on? I’m > curious, if it were implemented that way, would there ever be a need to > specify multiple MIBDIRS? > > On Mon, Nov 20, 2023 at 21:43 Ben K

  1   2   3   4   5   6   7   >