Hello @Brian I have a similar issue and I am seeing WAL Error, please take a look at the logs below. Can you please suggest one?
ts=2023-03-02T11:39:25.119Z caller=db.go:772 level=info component=tsdb msg="Found and deleted tmp block dir" dir=/data/01GTH1E28STFEJDTCCQVA8R813.tmp-for-creation ts=2023-03-02T11:39:25.120Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/data/lock ts=2023-03-02T11:39:32.713Z caller=head.go:493 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" ts=2023-03-02T11:39:38.194Z caller=head.go:527 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=5.480307055s ts=2023-03-02T11:39:38.194Z caller=head.go:533 level=info component=tsdb msg="Replaying WAL, this may take a while" On Thursday, August 18, 2022 at 2:18:27 PM UTC+5:30 Brian Candler wrote: > You'll need to show what happens in logs *after* that point (i.e. the > reloading of the WAL) - or does it freeze completely? > > Otherwise, what you're showing is normal error recovery. However it does > also suggest that there is something suspect about your storage. What sort > of storage are you using? Is it some remote NAS filesystem like NFS? If > so, that's not recommended. > > On Thursday, 18 August 2022 at 06:29:39 UTC+1 [email protected] wrote: > >> Hi, >> >> My Prometheus enviroment is working fine, until prometheus service was >> restarting automaticatly and stoping working. >> >> The error in my log file is: >> >> >> s=2022-08-12T17:06:30.837Z caller=repair.go:57 level=info component=tsdb >> msg="Found healthy block" mint=1660291200228 maxt=1660298400000 >> ulid=01GA8VC04PF4Z14NY2KWNJENET >> ts=2022-08-12T17:06:30.839Z caller=repair.go:57 level=info component=tsdb >> msg="Found healthy block" mint=1660262400229 maxt=1660284000000 >> ulid=01GA8VCCZVVXCW0JKVVZZM5GPE >> ts=2022-08-12T17:06:30.841Z caller=repair.go:57 level=info component=tsdb >> msg="Found healthy block" mint=1660298400000 maxt=1660305600000 >> ulid=01GA948VJW25YBS9PQJX0027ND >> ts=2022-08-12T17:06:30.851Z caller=db.go:777 level=info component=tsdb >> msg="Found and deleted tmp block dir" >> dir=data/01GA9GAZ99N1KF0SGEMT46RK2M.tmp-for-creation >> ts=2022-08-12T17:06:30.851Z caller=dir_locker.go:77 level=warn >> component=tsdb msg="A lockfile from a previous execution already existed. >> It was replaced" file=/data/lock >> ts=2022-08-12T17:06:31.451Z caller=head.go:493 level=info component=tsdb >> msg="Replaying on-disk memory mappable chunks if any" >> ts=2022-08-12T17:06:32.017Z caller=head.go:520 level=error component=tsdb >> msg="Loading on-disk chunks failed" err="iterate on on-disk chunks: out of >> sequence m-mapped chunk for series ref 2821831, last chunk: [1660312732989, >> 1660312792989], new: [1660312732989, 1660312792989]" >> ts=2022-08-12T17:06:32.019Z caller=head.go:689 level=info component=tsdb >> msg="Deleting mmapped chunk files" >> ts=2022-08-12T17:06:32.019Z caller=head.go:692 level=info component=tsdb >> msg="Deletion of corrupted mmap chunk files failed, discarding chunk files >> completely" err="cannot handle error: iterate on on-disk chunks: out of >> sequence m-mapped chunk for series ref 2821831, last chunk: [1660312732989, >> 1660312792989], new: [1660312732989, 1660312792989]" >> ts=2022-08-12T17:06:32.027Z caller=head.go:536 level=info component=tsdb >> msg="On-disk memory mappable chunks replay completed" duration=575.898846ms >> ts=2022-08-12T17:06:32.027Z caller=head.go:542 level=info component=tsdb >> msg="Replaying WAL, this may take a while" >> ts=2022-08-12T17:09:59.062Z caller=head.go:578 level=info component=tsdb >> msg="WAL checkpoint loaded" >> ts=2022-08-12T17:09:59.062Z caller=head.go:613 level=info component=tsdb >> msg="WAL segment loaded" segment=4645 maxSegment=4694 >> ts=2022-08-12T17:09:59.088Z caller=head.go:613 level=info component=tsdb >> msg="WAL segment loaded" segment=4646 maxSegment=4694 >> >> Any idea whats the problem? >> >> Thanks >> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e94cc96f-0961-402d-8ba1-783e032295fbn%40googlegroups.com.
