Thanks Ben, Its just come back from the dead without any intervention or changes.
Ok, I can sort the overlapping data, I think I know what that is. I don't believe we have limits on the Prometheus pod, it can do what it likes. With Prometheus running the metrics function within K8s, I can't see what the memory usage was at the time it failed. However at the moment its using about 14gb of Memory. The events on the pod show this: Readiness probe failed: Get "http://10.0.0.1:9090/-/ready": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Startup probe failed: HTTP probe failed with statuscode: 503 Liveness probe failed: Get "http://10.0.0.1:9090/-/healthy": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Back-off restarting failed container Liveness probe failed: Get "http://10.0.0.1:9090/-/healthy": context deadline exceeded (Client.Timeout exceeded while awaiting headers) No specific reason for the crash, just the health checks failing I guess. Does that still sound like memory to you? Thanks Nick On Tuesday, 23 May 2023 at 10:24:09 UTC+1 Ben Kochie wrote: > "Out of order sample from remote write". > > You have remote write clients that are sending out of order/overlapping > data. > > > https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus/ > > This is not likely to cause a crash IIRC. Have you checked to see if you > have allocated enough memory to your Prometheus? What does Kubernetes say > is the crash reason? > > In all probability, this is simply an OOM issue. Give your instance more > memory. > > On Tue, May 23, 2023 at 10:55 AM Nick Carlton <[email protected]> > wrote: > >> Hello, >> >> I have a Prometheus deployment in Azure using the kube-prometheus-stack >> helm chart which receives lots of remote writes. This runs fine normally >> but every couple of months or so, will randomly crash and start this crash >> loop cycle. It does usually self-resolve but takes Prom out for a couple of >> days. >> >> We use a PVC with Prometheus with a size of 200GB and with the access >> mode "ReadWriteOnce". >> >> Basically what I am seeing happen is, Prometheus Starts, complains about >> some storage chunks and does a WAL replay. Then it does start, however, we >> get thousands of messages about remote write "out of bounds" and then it >> crashes and loops again. Its will continue like this even if the pods are >> deleted and re-created. >> >> See Logs: >> >> ts=2023-05-23T08:12:03.968Z caller=main.go:556 level=info msg="Starting >> Prometheus Server" mode=server version="(version=2.42.0, branch=HEAD, >> revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" >> ts=2023-05-23T08:12:03.968Z caller=main.go:561 level=info >> build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, >> date=20230201-07:53:32)" ts=2023-05-23T08:12:03.968Z caller=main.go:562 >> level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu >> May 19 18:52:16 UTC 2022 x86_64 >> prometheus-prometheus-kube-prometheus-prometheus-0 (none))" >> ts=2023-05-23T08:12:03.968Z caller=main.go:563 level=info >> fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:12:03.968Z >> caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" >> ts=2023-05-23T08:12:03.970Z caller=web.go:561 level=info component=web >> msg="Start listening for connections" address=0.0.0.0:9090 >> ts=2023-05-23T08:12:03.970Z caller=main.go:993 level=info msg="Starting >> TSDB ..." ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info >> component=tsdb msg="Found healthy block" mint=1683892800000 >> maxt=1683957600000 ulid=01H0A5J8MWQ152H85DYY6F32PK >> ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1683957600000 maxt=1684022400000 >> ulid=01H0C38Y02M29M82TXK2V7NN7G ts=2023-05-23T08:12:03.970Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684022400000 maxt=1684087200000 ulid=01H0E12DTC6PWH4J6R7B1H36XW >> ts=2023-05-23T08:12:03.970Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684087200003 maxt=1684152000000 >> ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684152000003 maxt=1684216800000 ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684216800000 maxt=1684281600000 >> ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684281600003 maxt=1684346400000 ulid=01H0NZ78AACK9BPD034VWH7SJS >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684346400000 maxt=1684411200000 >> ulid=01H0QP2AA05C33ZVAAVDBT2E43 ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684411200004 maxt=1684476000000 ulid=01H0STTM76GGEKKYAMWV1KNTPT >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684476000004 maxt=1684540800000 >> ulid=01H0VRMBDAC5D5M6TEVHRP80PS ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684540800043 maxt=1684605600000 ulid=01H0XPDYZ3D2X8XJ97B97M4BDH >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684605600043 maxt=1684670400000 >> ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684735200044 maxt=1684742400000 ulid=01H11AZP33QHEGWR2631MYMY3M >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684742400044 maxt=1684749600000 >> ulid=01H11HVD455FTTADM149WAHAH0 ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684670400032 maxt=1684735200000 ulid=01H11J11TMN4XWACE1Q9AJ1KXC >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684749600043 maxt=1684756800000 >> ulid=01H11RQ4AHY8PVQRCM58CKTVZF ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684756800043 maxt=1684764000000 ulid=01H11ZJVV4E276NWR3370HFN00 >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684764000000 maxt=1684771200000 >> ulid=01H12712JWFQMYW8FR8YXGQN8Y ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684771200000 maxt=1684778400000 ulid=01H12DAKVHHHGEY3TKY0FPQSVZ >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684778400013 maxt=1684785600000 >> ulid=01H12M7CW13EEQRVRN6J4ZD7AK ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684785600070 maxt=1684792800000 ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684792800033 maxt=1684800000000 >> ulid=01H131XPKE4VB6RMRQF3S67M0Y ts=2023-05-23T08:12:03.971Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684800000010 maxt=1684807200000 ulid=01H138S7B8Y8Q14YKJZ5SRXTAX >> ts=2023-05-23T08:12:03.971Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684807200000 maxt=1684814400000 >> ulid=01H13FXP6EB7T1KSRR8P5GMFP3 ts=2023-05-23T08:12:03.971Z >> caller=tls_config.go:232 level=info component=web msg="Listening on" >> address=[::]:9090 ts=2023-05-23T08:12:03.971Z caller=repair.go:56 >> level=info component=tsdb msg="Found healthy block" mint=1684814400043 >> maxt=1684821600000 ulid=01H13PQXNNJNH4GETATBWHPGYJ >> ts=2023-05-23T08:12:03.971Z caller=tls_config.go:271 level=info >> component=web msg="TLS is disabled." http2=false address=[::]:9090 >> ts=2023-05-23T08:12:04.257Z caller=head.go:564 level=info component=tsdb >> msg="Replaying on-disk memory mappable chunks if any" >> ts=2023-05-23T08:12:04.318Z caller=head.go:592 level=error component=tsdb >> msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of >> sequence m-mapped chunk for series ref 19360078, last chunk: >> [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" >> ts=2023-05-23T08:12:04.318Z caller=head.go:831 level=info component=tsdb >> msg="Deleting mmapped chunk files" ts=2023-05-23T08:12:04.319Z >> caller=head.go:837 level=info component=tsdb msg="Deleting mmapped chunk >> files" ts=2023-05-23T08:12:04.319Z caller=head.go:840 level=info >> component=tsdb msg="Deletion of corrupted mmap chunk files failed, >> discarding chunk files completely" err="cannot handle error: iterate on >> on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, >> last chunk: [1684824583256, 1684825285256], new: [1684822459256, >> 1684823161256]" ts=2023-05-23T08:12:04.325Z caller=head.go:608 level=info >> component=tsdb msg="On-disk memory mappable chunks replay completed" >> duration=67.431625ms ts=2023-05-23T08:12:04.325Z caller=head.go:614 >> level=info component=tsdb msg="Replaying WAL, this may take a while" >> ts=2023-05-23T08:12:14.136Z caller=head.go:650 level=info component=tsdb >> msg="WAL checkpoint loaded" ts=2023-05-23T08:12:14.145Z caller=head.go:685 >> level=info component=tsdb msg="WAL segment loaded" segment=17588 >> maxSegment=17630 ts=2023-05-23T08:12:14.197Z caller=head.go:685 level=info >> component=tsdb msg="WAL segment loaded" segment=17589 maxSegment=17630 >> ts=2023-05-23T08:12:14.216Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17590 maxSegment=17630 >> ts=2023-05-23T08:12:14.250Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17591 maxSegment=17630 >> ts=2023-05-23T08:12:14.293Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17592 maxSegment=17630 >> ts=2023-05-23T08:12:15.026Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17593 maxSegment=17630 >> ts=2023-05-23T08:12:15.063Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17594 maxSegment=17630 >> ts=2023-05-23T08:12:15.188Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17595 maxSegment=17630 >> ts=2023-05-23T08:12:15.287Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17596 maxSegment=17630 >> ts=2023-05-23T08:12:15.378Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17597 maxSegment=17630 >> ts=2023-05-23T08:12:15.457Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17598 maxSegment=17630 >> ts=2023-05-23T08:12:15.543Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17599 maxSegment=17630 >> ts=2023-05-23T08:12:15.619Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17600 maxSegment=17630 >> ts=2023-05-23T08:12:15.745Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17601 maxSegment=17630 >> ts=2023-05-23T08:12:15.890Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17602 maxSegment=17630 >> ts=2023-05-23T08:12:15.977Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17603 maxSegment=17630 >> ts=2023-05-23T08:12:16.057Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17604 maxSegment=17630 >> ts=2023-05-23T08:12:16.156Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17605 maxSegment=17630 >> ts=2023-05-23T08:12:16.157Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17606 maxSegment=17630 >> ts=2023-05-23T08:12:17.091Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17607 maxSegment=17630 >> ts=2023-05-23T08:12:18.271Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17608 maxSegment=17630 >> ts=2023-05-23T08:12:18.449Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17609 maxSegment=17630 >> ts=2023-05-23T08:12:18.787Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17610 maxSegment=17630 >> ts=2023-05-23T08:12:19.671Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17611 maxSegment=17630 >> ts=2023-05-23T08:12:19.894Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17612 maxSegment=17630 >> ts=2023-05-23T08:12:20.178Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17613 maxSegment=17630 >> ts=2023-05-23T08:12:20.344Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17614 maxSegment=17630 >> ts=2023-05-23T08:12:22.299Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17616 maxSegment=17630 >> ts=2023-05-23T08:12:23.180Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17617 maxSegment=17630 >> ts=2023-05-23T08:12:23.343Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17618 maxSegment=17630 >> ts=2023-05-23T08:12:23.599Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17619 maxSegment=17630 >> ts=2023-05-23T08:12:23.764Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17620 maxSegment=17630 >> ts=2023-05-23T08:12:24.128Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17621 maxSegment=17630 >> ts=2023-05-23T08:12:24.317Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17622 maxSegment=17630 >> ts=2023-05-23T08:12:24.463Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17623 maxSegment=17630 >> ts=2023-05-23T08:12:24.809Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17624 maxSegment=17630 >> ts=2023-05-23T08:12:25.088Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17625 maxSegment=17630 >> ts=2023-05-23T08:12:25.360Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17626 maxSegment=17630 >> ts=2023-05-23T08:12:25.642Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17627 maxSegment=17630 >> ts=2023-05-23T08:12:25.664Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17628 maxSegment=17630 >> ts=2023-05-23T08:12:25.884Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17629 maxSegment=17630 >> ts=2023-05-23T08:12:25.884Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17630 maxSegment=17630 >> ts=2023-05-23T08:12:25.884Z caller=head.go:722 level=info component=tsdb >> msg="WAL replay completed" checkpoint_replay_duration=9.811449959s >> wal_replay_duration=11.747824132s wbl_replay_duration=215ns >> total_replay_duration=21.626738156s ts=2023-05-23T08:12:28.177Z >> caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC >> ts=2023-05-23T08:12:28.177Z caller=main.go:1017 level=info msg="TSDB >> started" ts=2023-05-23T08:12:28.177Z caller=main.go:1197 level=info >> msg="Loading configuration file" >> filename=/etc/prometheus/config_out/prometheus.env.yaml >> ts=2023-05-23T08:12:28.181Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-operator/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:12:28.182Z caller=kubernetes.go:326 level=info >> component="discovery manager notify" discovery=kubernetes config=config-0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:12:28.221Z caller=main.go:1234 level=info msg="Completed >> loading of configuration file" >> filename=/etc/prometheus/config_out/prometheus.env.yaml >> totalDuration=43.976379ms db_storage=1.549µs remote_storage=1.501µs >> web_handler=594ns query_engine=1.035µs scrape=207.016µs scrape_sd=565.387µs >> notify=13.403µs notify_sd=116.569µs rules=39.2593ms tracing=5.19µs >> ts=2023-05-23T08:12:28.221Z caller=main.go:978 level=info msg="Server is >> ready to receive web requests." ts=2023-05-23T08:12:28.221Z >> caller=manager.go:974 level=info component="rule manager" msg="Starting >> rule manager..." ts=2023-05-23T08:13:17.453Z caller=write_handler.go:109 >> level=error component=web msg="Out of order sample from remote write" >> err="out of bounds" series ... timestamp=1684825890472 ... >> ts=2023-05-23T08:13:17.547Z caller=write_handler.go:109 level=error >> component=web msg="Out of order sample from remote write" err="out of >> bounds" series ... timestamp=1684808929971 ts=2023-05-23T08:13:29.090Z >> caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server >> version="(version=2.42.0, branch=HEAD, >> revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" >> ts=2023-05-23T08:13:29.090Z caller=main.go:561 level=info >> build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, >> date=20230201-07:53:32)" ts=2023-05-23T08:13:29.090Z caller=main.go:562 >> level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu >> May 19 18:52:16 UTC 2022 x86_64 >> prometheus-prometheus-kube-prometheus-prometheus-0 (none))" >> ts=2023-05-23T08:13:29.090Z caller=main.go:563 level=info >> fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:13:29.090Z >> caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" >> ts=2023-05-23T08:13:29.090Z caller=query_logger.go:80 level=info >> component=activeQueryTracker msg="These queries didn't finish in >> prometheus' last run:" queries=[829607] ts=2023-05-23T08:13:29.091Z >> caller=web.go:561 level=info component=web msg="Start listening for >> connections" address=0.0.0.0:9090 ts=2023-05-23T08:13:29.092Z >> caller=main.go:993 level=info msg="Starting TSDB ..." >> ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1683892800000 maxt=1683957600000 >> ulid=01H0A5J8MWQ152H85DYY6F32PK ts=2023-05-23T08:13:29.092Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1683957600000 maxt=1684022400000 ulid=01H0C38Y02M29M82TXK2V7NN7G >> ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684022400000 maxt=1684087200000 >> ulid=01H0E12DTC6PWH4J6R7B1H36XW ts=2023-05-23T08:13:29.092Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684087200003 maxt=1684152000000 ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ >> ts=2023-05-23T08:13:29.092Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684152000003 maxt=1684216800000 >> ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684216800000 maxt=1684281600000 ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684281600003 maxt=1684346400000 >> ulid=01H0NZ78AACK9BPD034VWH7SJS ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684346400000 maxt=1684411200000 ulid=01H0QP2AA05C33ZVAAVDBT2E43 >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684411200004 maxt=1684476000000 >> ulid=01H0STTM76GGEKKYAMWV1KNTPT ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684476000004 maxt=1684540800000 ulid=01H0VRMBDAC5D5M6TEVHRP80PS >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684540800043 maxt=1684605600000 >> ulid=01H0XPDYZ3D2X8XJ97B97M4BDH ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684605600043 maxt=1684670400000 ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 >> ts=2023-05-23T08:13:29.093Z caller=tls_config.go:232 level=info >> component=web msg="Listening on" address=[::]:9090 >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684735200044 maxt=1684742400000 >> ulid=01H11AZP33QHEGWR2631MYMY3M ts=2023-05-23T08:13:29.093Z >> caller=tls_config.go:271 level=info component=web msg="TLS is disabled." >> http2=false address=[::]:9090 ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684742400044 maxt=1684749600000 ulid=01H11HVD455FTTADM149WAHAH0 >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684670400032 maxt=1684735200000 >> ulid=01H11J11TMN4XWACE1Q9AJ1KXC ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684749600043 maxt=1684756800000 ulid=01H11RQ4AHY8PVQRCM58CKTVZF >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684756800043 maxt=1684764000000 >> ulid=01H11ZJVV4E276NWR3370HFN00 ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684764000000 maxt=1684771200000 ulid=01H12712JWFQMYW8FR8YXGQN8Y >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684771200000 maxt=1684778400000 >> ulid=01H12DAKVHHHGEY3TKY0FPQSVZ ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684778400013 maxt=1684785600000 ulid=01H12M7CW13EEQRVRN6J4ZD7AK >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684785600070 maxt=1684792800000 >> ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684792800033 maxt=1684800000000 ulid=01H131XPKE4VB6RMRQF3S67M0Y >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684800000010 maxt=1684807200000 >> ulid=01H138S7B8Y8Q14YKJZ5SRXTAX ts=2023-05-23T08:13:29.093Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684807200000 maxt=1684814400000 ulid=01H13FXP6EB7T1KSRR8P5GMFP3 >> ts=2023-05-23T08:13:29.093Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684814400043 maxt=1684821600000 >> ulid=01H13PQXNNJNH4GETATBWHPGYJ ts=2023-05-23T08:13:29.365Z >> caller=head.go:564 level=info component=tsdb msg="Replaying on-disk memory >> mappable chunks if any" ts=2023-05-23T08:13:29.426Z caller=head.go:608 >> level=info component=tsdb msg="On-disk memory mappable chunks replay >> completed" duration=60.526867ms ts=2023-05-23T08:13:29.426Z >> caller=head.go:614 level=info component=tsdb msg="Replaying WAL, this may >> take a while" ts=2023-05-23T08:13:38.994Z caller=head.go:650 level=info >> component=tsdb msg="WAL checkpoint loaded" ts=2023-05-23T08:13:39.003Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17588 maxSegment=17631 ts=2023-05-23T08:13:39.038Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17589 maxSegment=17631 ts=2023-05-23T08:13:39.055Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17590 maxSegment=17631 ts=2023-05-23T08:13:39.081Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17591 maxSegment=17631 ts=2023-05-23T08:13:39.120Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17592 maxSegment=17631 ts=2023-05-23T08:13:40.778Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17593 maxSegment=17631 ts=2023-05-23T08:13:40.811Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17594 maxSegment=17631 ts=2023-05-23T08:13:40.928Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17595 maxSegment=17631 ts=2023-05-23T08:13:41.015Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17596 maxSegment=17631 ts=2023-05-23T08:13:41.091Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17597 maxSegment=17631 ts=2023-05-23T08:13:41.158Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17598 maxSegment=17631 ts=2023-05-23T08:13:41.234Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17599 maxSegment=17631 ts=2023-05-23T08:13:41.300Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17600 maxSegment=17631 ts=2023-05-23T08:13:41.406Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17601 maxSegment=17631 ts=2023-05-23T08:13:41.543Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17602 maxSegment=17631 ts=2023-05-23T08:13:41.619Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17603 maxSegment=17631 ts=2023-05-23T08:13:41.689Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17604 maxSegment=17631 ts=2023-05-23T08:13:41.770Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17605 maxSegment=17631 ts=2023-05-23T08:13:41.771Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17606 maxSegment=17631 ts=2023-05-23T08:13:42.359Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17607 maxSegment=17631 ts=2023-05-23T08:13:42.843Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17608 maxSegment=17631 ts=2023-05-23T08:13:43.023Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17609 maxSegment=17631 ts=2023-05-23T08:13:43.362Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17610 maxSegment=17631 ts=2023-05-23T08:13:44.268Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17611 maxSegment=17631 ts=2023-05-23T08:13:44.500Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17612 maxSegment=17631 ts=2023-05-23T08:13:44.792Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17613 maxSegment=17631 ts=2023-05-23T08:13:44.958Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17614 maxSegment=17631 ts=2023-05-23T08:13:45.175Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17615 maxSegment=17631 ts=2023-05-23T08:13:46.881Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17616 maxSegment=17631 ts=2023-05-23T08:13:48.160Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17617 maxSegment=17631 ts=2023-05-23T08:13:48.529Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17618 maxSegment=17631 ts=2023-05-23T08:13:49.071Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17619 maxSegment=17631 ts=2023-05-23T08:13:49.463Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17620 maxSegment=17631 ts=2023-05-23T08:13:49.864Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17621 maxSegment=17631 ts=2023-05-23T08:13:50.052Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17622 maxSegment=17631 ts=2023-05-23T08:13:50.181Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17623 maxSegment=17631 ts=2023-05-23T08:13:51.235Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17627 maxSegment=17631 ts=2023-05-23T08:13:51.258Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17628 maxSegment=17631 ts=2023-05-23T08:13:51.476Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17629 maxSegment=17631 ts=2023-05-23T08:13:51.707Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17630 maxSegment=17631 ts=2023-05-23T08:13:51.707Z >> caller=head.go:685 level=info component=tsdb msg="WAL segment loaded" >> segment=17631 maxSegment=17631 ts=2023-05-23T08:13:51.707Z >> caller=head.go:722 level=info component=tsdb msg="WAL replay completed" >> checkpoint_replay_duration=9.568658965s wal_replay_duration=12.712461039s >> wbl_replay_duration=240ns total_replay_duration=22.341708746s >> ts=2023-05-23T08:13:54.095Z caller=main.go:1014 level=info >> fs_type=EXT4_SUPER_MAGIC ts=2023-05-23T08:13:54.095Z caller=main.go:1017 >> level=info msg="TSDB started" ts=2023-05-23T08:13:54.095Z >> caller=main.go:1197 level=info msg="Loading configuration file" >> filename=/etc/prometheus/config_out/prometheus.env.yaml >> ts=2023-05-23T08:13:54.098Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-kubelet/2 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-prometheus/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:13:54.099Z caller=kubernetes.go:326 level=info >> component="discovery manager notify" discovery=kubernetes config=config-0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:13:54.136Z caller=main.go:1234 level=info msg="Completed >> loading of configuration file" >> filename=/etc/prometheus/config_out/prometheus.env.yaml >> totalDuration=41.309545ms db_storage=946ns remote_storage=1.338µs >> web_handler=633ns query_engine=1.422µs scrape=229.86µs scrape_sd=593.335µs >> notify=15.321µs notify_sd=186.832µs rules=36.799825ms tracing=4.126µs >> ts=2023-05-23T08:13:54.136Z caller=main.go:978 level=info msg="Server is >> ready to receive web requests." ts=2023-05-23T08:13:54.136Z >> caller=manager.go:974 level=info component="rule manager" msg="Starting >> rule manager..." ts=2023-05-23T08:13:58.936Z caller=write_handler.go:109 >> level=error component=web msg="Out of order sample from remote write" >> err="out of bounds" series ... timestamp=1684808929971 ... >> ts=2023-05-23T08:14:16.023Z caller=write_handler.go:109 level=error >> component=web msg="Out of order sample from remote write" err="out of >> bounds" series ... timestamp=1684825987289 ts=2023-05-23T08:14:34.749Z >> caller=main.go:556 level=info msg="Starting Prometheus Server" mode=server >> version="(version=2.42.0, branch=HEAD, >> revision=225c61122d88b01d1f0eaaee0e05b6f3e0567ac0)" >> ts=2023-05-23T08:14:34.749Z caller=main.go:561 level=info >> build_context="(go=go1.19.5, platform=linux/amd64, user=root@c67d48967507, >> date=20230201-07:53:32)" ts=2023-05-23T08:14:34.749Z caller=main.go:562 >> level=info host_details="(Linux 5.4.0-1080-azure #83~18.04.2-Ubuntu SMP Thu >> May 19 18:52:16 UTC 2022 x86_64 >> prometheus-prometheus-kube-prometheus-prometheus-0 (none))" >> ts=2023-05-23T08:14:34.749Z caller=main.go:563 level=info >> fd_limits="(soft=1048576, hard=1048576)" ts=2023-05-23T08:14:34.749Z >> caller=main.go:564 level=info vm_limits="(soft=unlimited, hard=unlimited)" >> ts=2023-05-23T08:14:34.750Z caller=web.go:561 level=info component=web >> msg="Start listening for connections" address=0.0.0.0:9090 >> ts=2023-05-23T08:14:34.752Z caller=main.go:993 level=info msg="Starting >> TSDB ..." ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info >> component=tsdb msg="Found healthy block" mint=1683892800000 >> maxt=1683957600000 ulid=01H0A5J8MWQ152H85DYY6F32PK >> ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1683957600000 maxt=1684022400000 >> ulid=01H0C38Y02M29M82TXK2V7NN7G ts=2023-05-23T08:14:34.752Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684022400000 maxt=1684087200000 ulid=01H0E12DTC6PWH4J6R7B1H36XW >> ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684087200003 maxt=1684152000000 >> ulid=01H0G5TJ4RCKN0T7ZGSAJNCCVJ ts=2023-05-23T08:14:34.752Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684152000003 maxt=1684216800000 ulid=01H0J3M6H3Q3G4EK3F0JA5XGX6 >> ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684216800000 maxt=1684281600000 >> ulid=01H0KTF3N2VYP9DT5HZC2DPMJ8 ts=2023-05-23T08:14:34.752Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684281600003 maxt=1684346400000 ulid=01H0NZ78AACK9BPD034VWH7SJS >> ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684346400000 maxt=1684411200000 >> ulid=01H0QP2AA05C33ZVAAVDBT2E43 ts=2023-05-23T08:14:34.752Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684411200004 maxt=1684476000000 ulid=01H0STTM76GGEKKYAMWV1KNTPT >> ts=2023-05-23T08:14:34.752Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684476000004 maxt=1684540800000 >> ulid=01H0VRMBDAC5D5M6TEVHRP80PS ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684540800043 maxt=1684605600000 ulid=01H0XPDYZ3D2X8XJ97B97M4BDH >> ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684605600043 maxt=1684670400000 >> ulid=01H0ZM6Y7X90Q4D5YR9G1BJ3F4 ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684735200044 maxt=1684742400000 ulid=01H11AZP33QHEGWR2631MYMY3M >> ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684742400044 maxt=1684749600000 >> ulid=01H11HVD455FTTADM149WAHAH0 ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684670400032 maxt=1684735200000 ulid=01H11J11TMN4XWACE1Q9AJ1KXC >> ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684749600043 maxt=1684756800000 >> ulid=01H11RQ4AHY8PVQRCM58CKTVZF ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684756800043 maxt=1684764000000 ulid=01H11ZJVV4E276NWR3370HFN00 >> ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684764000000 maxt=1684771200000 >> ulid=01H12712JWFQMYW8FR8YXGQN8Y ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684771200000 maxt=1684778400000 ulid=01H12DAKVHHHGEY3TKY0FPQSVZ >> ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684778400013 maxt=1684785600000 >> ulid=01H12M7CW13EEQRVRN6J4ZD7AK ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684785600070 maxt=1684792800000 ulid=01H12V1ZBEM4BGA5ZNRQRVT9XM >> ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684792800033 maxt=1684800000000 >> ulid=01H131XPKE4VB6RMRQF3S67M0Y ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684800000010 maxt=1684807200000 ulid=01H138S7B8Y8Q14YKJZ5SRXTAX >> ts=2023-05-23T08:14:34.753Z caller=repair.go:56 level=info component=tsdb >> msg="Found healthy block" mint=1684807200000 maxt=1684814400000 >> ulid=01H13FXP6EB7T1KSRR8P5GMFP3 ts=2023-05-23T08:14:34.753Z >> caller=repair.go:56 level=info component=tsdb msg="Found healthy block" >> mint=1684814400043 maxt=1684821600000 ulid=01H13PQXNNJNH4GETATBWHPGYJ >> ts=2023-05-23T08:14:34.753Z caller=tls_config.go:232 level=info >> component=web msg="Listening on" address=[::]:9090 >> ts=2023-05-23T08:14:34.753Z caller=tls_config.go:271 level=info >> component=web msg="TLS is disabled." http2=false address=[::]:9090 >> ts=2023-05-23T08:14:35.063Z caller=head.go:564 level=info component=tsdb >> msg="Replaying on-disk memory mappable chunks if any" >> ts=2023-05-23T08:14:35.125Z caller=head.go:592 level=error component=tsdb >> msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of >> sequence m-mapped chunk for series ref 19360078, last chunk: >> [1684824583256, 1684825285256], new: [1684822459256, 1684823161256]" >> ts=2023-05-23T08:14:35.125Z caller=head.go:831 level=info component=tsdb >> msg="Deleting mmapped chunk files" ts=2023-05-23T08:14:35.129Z >> caller=head.go:837 level=info component=tsdb msg="Deleting mmapped chunk >> files" ts=2023-05-23T08:14:35.129Z caller=head.go:840 level=info >> component=tsdb msg="Deletion of corrupted mmap chunk files failed, >> discarding chunk files completely" err="cannot handle error: iterate on >> on-disk chunks: out of sequence m-mapped chunk for series ref 19360078, >> last chunk: [1684824583256, 1684825285256], new: [1684822459256, >> 1684823161256]" ts=2023-05-23T08:14:35.135Z caller=head.go:608 level=info >> component=tsdb msg="On-disk memory mappable chunks replay completed" >> duration=71.923626ms ts=2023-05-23T08:14:35.135Z caller=head.go:614 >> level=info component=tsdb msg="Replaying WAL, this may take a while" >> ts=2023-05-23T08:14:44.651Z caller=head.go:650 level=info component=tsdb >> msg="WAL checkpoint loaded" ts=2023-05-23T08:14:44.659Z caller=head.go:685 >> level=info component=tsdb msg="WAL segment loaded" segment=17588 >> maxSegment=17632 ts=2023-05-23T08:14:44.710Z caller=head.go:685 level=info >> component=tsdb msg="WAL segment loaded" segment=17589 maxSegment=17632 >> ts=2023-05-23T08:14:44.729Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17590 maxSegment=17632 >> ts=2023-05-23T08:14:44.763Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17591 maxSegment=17632 >> ts=2023-05-23T08:14:44.894Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17592 maxSegment=17632 >> ts=2023-05-23T08:14:46.470Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17593 maxSegment=17632 >> ts=2023-05-23T08:14:46.503Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17594 maxSegment=17632 >> ts=2023-05-23T08:14:46.616Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17595 maxSegment=17632 >> ts=2023-05-23T08:14:46.710Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17596 maxSegment=17632 >> ts=2023-05-23T08:14:46.796Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17597 maxSegment=17632 >> ts=2023-05-23T08:14:46.876Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17598 maxSegment=17632 >> ts=2023-05-23T08:14:46.955Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17599 maxSegment=17632 >> ts=2023-05-23T08:14:47.025Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17600 maxSegment=17632 >> ts=2023-05-23T08:14:47.135Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17601 maxSegment=17632 >> ts=2023-05-23T08:14:47.252Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17602 maxSegment=17632 >> ts=2023-05-23T08:14:47.336Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17603 maxSegment=17632 >> ts=2023-05-23T08:14:47.406Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17604 maxSegment=17632 >> ts=2023-05-23T08:14:47.496Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17605 maxSegment=17632 >> ts=2023-05-23T08:14:47.499Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17606 maxSegment=17632 >> ts=2023-05-23T08:14:48.079Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17607 maxSegment=17632 >> ts=2023-05-23T08:14:48.573Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17608 maxSegment=17632 >> ts=2023-05-23T08:14:48.745Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17609 maxSegment=17632 >> ts=2023-05-23T08:14:49.082Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17610 maxSegment=17632 >> ts=2023-05-23T08:14:49.967Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17611 maxSegment=17632 >> ts=2023-05-23T08:14:50.190Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17612 maxSegment=17632 >> ts=2023-05-23T08:14:52.558Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17616 maxSegment=17632 >> ts=2023-05-23T08:14:54.144Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17617 maxSegment=17632 >> ts=2023-05-23T08:14:54.490Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17618 maxSegment=17632 >> ts=2023-05-23T08:14:55.119Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17619 maxSegment=17632 >> ts=2023-05-23T08:14:55.293Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17620 maxSegment=17632 >> ts=2023-05-23T08:14:55.633Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17621 maxSegment=17632 >> ts=2023-05-23T08:14:55.826Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17622 maxSegment=17632 >> ts=2023-05-23T08:14:55.976Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17623 maxSegment=17632 >> ts=2023-05-23T08:14:56.335Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17624 maxSegment=17632 >> ts=2023-05-23T08:14:56.615Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17625 maxSegment=17632 >> ts=2023-05-23T08:14:56.882Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17626 maxSegment=17632 >> ts=2023-05-23T08:14:57.171Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17627 maxSegment=17632 >> ts=2023-05-23T08:14:57.193Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17628 maxSegment=17632 >> ts=2023-05-23T08:14:57.402Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17629 maxSegment=17632 >> ts=2023-05-23T08:14:57.616Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17630 maxSegment=17632 >> ts=2023-05-23T08:14:57.718Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17631 maxSegment=17632 >> ts=2023-05-23T08:14:57.718Z caller=head.go:685 level=info component=tsdb >> msg="WAL segment loaded" segment=17632 maxSegment=17632 >> ts=2023-05-23T08:14:57.718Z caller=head.go:722 level=info component=tsdb >> msg="WAL replay completed" checkpoint_replay_duration=9.515250193s >> wal_replay_duration=13.067369985s wbl_replay_duration=205ns >> total_replay_duration=22.654583208s ts=2023-05-23T08:15:00.133Z >> caller=main.go:1014 level=info fs_type=EXT4_SUPER_MAGIC >> ts=2023-05-23T08:15:00.133Z caller=main.go:1017 level=info msg="TSDB >> started" ts=2023-05-23T08:15:00.134Z caller=main.go:1197 level=info >> msg="Loading configuration file" >> filename=/etc/prometheus/config_out/prometheus.env.yaml >> ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-alertmanager/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-apiserver/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info >> component="discovery manager scrape" discovery=kubernetes >> config=serviceMonitor/monitoring/prometheus-kube-prometheus-kube-scheduler/0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:15:00.138Z caller=kubernetes.go:326 level=info >> component="discovery manager notify" discovery=kubernetes config=config-0 >> msg="Using pod service account via in-cluster config" >> ts=2023-05-23T08:15:00.176Z caller=main.go:1234 level=info msg="Completed >> loading of configuration file" >> filename=/etc/prometheus/config_out/prometheus.env.yaml >> totalDuration=42.962628ms db_storage=1.094µs remote_storage=1.525µs >> web_handler=583ns query_engine=1.243µs scrape=197.887µs scrape_sd=650.383µs >> notify=16.671µs notify_sd=181.635µs rules=38.073939ms tracing=4.418µs >> ts=2023-05-23T08:15:00.177Z caller=main.go:978 level=info msg="Server is >> ready to receive web requests." ts=2023-05-23T08:15:00.177Z >> caller=manager.go:974 level=info component="rule manager" msg="Starting >> rule manager..." >> >> >> Sometimes Prom will just start up again without warning and other times I >> see this after all the out of bounds errors: >> >> ts=2023-05-23T08:18:58.284Z caller=main.go:828 level=warn msg="Received >> SIGTERM, exiting gracefully..." ts=2023-05-23T08:18:58.284Z >> caller=main.go:852 level=info msg="Stopping scrape discovery manager..." >> ts=2023-05-23T08:18:58.284Z caller=main.go:866 level=info msg="Stopping >> notify discovery manager..." ts=2023-05-23T08:18:58.284Z >> caller=manager.go:988 level=info component="rule manager" msg="Stopping >> rule manager..." ts=2023-05-23T08:18:58.284Z caller=main.go:848 level=info >> msg="Scrape discovery manager stopped" ts=2023-05-23T08:18:58.284Z >> caller=main.go:862 level=info msg="Notify discovery manager stopped" >> >> Is anyone able to assist with what is going wrong here? >> >> Thanks >> Nick >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/ab9d7a7f-b4d8-49ee-b702-fda49a590102n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/ab9d7a7f-b4d8-49ee-b702-fda49a590102n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f1b3ce29-ed3e-4c13-a973-661c646414b0n%40googlegroups.com.
