Hello,
Metrics are not saved when I put a timestamps with metrics.
I've got through github issue and some of prometheus codes but not yet
getting the root cause.
I would be happy if I can get why I'm getting below errors.
Errors
*Error 1*
I got below error when executing prometheus with prometheus.yml (v1).
Please see blow for the prometheus.yml (v1).
prometheus_1 | level=warn ts=2020-05-25T12:29:17.216Z caller=scrape.go:1240
component="scrape manager" scrape_pool=my-app target=https://<internal
server address>:443/metrics msg="Error on ingesting samples that are too
old or are too far into the future" num_dropped=600
*Error2*
I got massive amount of below log when running prometheus with
prometheus.yml (v2). Please see below for prometheus.yml (v2)
prometheus_1 | level=info ts=2020-05-25T15:57:25.857Z caller=head.go:732
component=tsdb msg="WAL checkpoint complete" first=1164 last=1165
duration=2.1818ms
prometheus_1 | level=info ts=2020-05-25T15:57:25.867Z caller=head.go:662
component=tsdb msg="head GC completed" duration=509.9µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.879Z caller=head.go:662
component=tsdb msg="head GC completed" duration=549.7µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.882Z caller=head.go:732
component=tsdb msg="WAL checkpoint complete" first=1166 last=1167
duration=2.7017ms
prometheus_1 | level=info ts=2020-05-25T15:57:25.892Z caller=head.go:662
component=tsdb msg="head GC completed" duration=564.7µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.903Z caller=head.go:662
component=tsdb msg="head GC completed" duration=651.9µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.906Z caller=head.go:732
component=tsdb msg="WAL checkpoint complete" first=1168 last=1169
duration=2.8861ms
prometheus_1 | level=info ts=2020-05-25T15:57:25.916Z caller=head.go:662
component=tsdb msg="head GC completed" duration=562.8µs
prometheus_1 | level=info ts=2020-05-25T15:57:25.925Z caller=head.go:662
component=tsdb msg="head GC completed" duration=633µs
Background
I'm testing prometheus for custom app.
I've created endpoint which expose metrics in text-based format for the app.
Since metrics are calculated with batch job, I've put a timestamps to
metrics to show when the metrics was calculated.
The batch job is executed in every 5 min, so the timestamp will not be
older than 5 min compared to the current timestamp.
Here is an example of exposed data. There are about 600 lines in real data.
#HELP processed_rows_total the total number of processed rows
#TYPE processed_rows_total counter
processed_rows_total{stream_name="a", result="success"} 93163 1590422899
processed_rows_total{stream_name="a", result="error"} 0 1590422899
processed_rows_total{stream_name="b", result="success"} 45695851 1590422899
processed_rows_total{stream_name="b", result="error"} 0 1590422899
Environment
Running with docker.
docker-compose.yml
prometheus:
image: prom/prometheus:v2.17.2
entrypoint:
- "/bin/prometheus"
- "--log.level=debug"
- "--config.file=/etc/prometheus/prometheus.yml"
- "--storage.tsdb.path=/prometheus"
- "--web.console.libraries=/usr/share/prometheus/console_libraries"
- "--web.console.templates=/usr/share/prometheus/consoles"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
ports:
- "127.0.0.1:9090:9090"
prometheus.yml (v1)
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds.
Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default
is every 1 minute.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries
scraped from this config.
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'my-app'
metrics_path: '/metrics'
scrape_interval: 1m
scheme: 'https'
static_configs:
- targets: ['<internal server address>']
prometheus.yml (v2)
# my global config
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds.
Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default
is every 1 minute.
# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
- job_name: 'my-app'
metrics_path: '/metrics'
scrape_interval: 1m
scheme: 'https'
static_configs:
- targets: ['<internal server address>']
Other notes
With *honor-timestamps: false* configuration in prometheus.yml, metrics are
saved correctly.
--
You received this message because you are subscribed to the Google Groups
"Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/prometheus-users/64c8a462-5130-4747-a1a5-49e67b975aa3%40googlegroups.com.
