Hey, I am still getting the same issue. Anymore suggestions?
On Thursday, 5 October 2023 at 11:24:17 UTC-7 Anoop Mohan wrote: > Hi Ben, > > I have not configured the CPU/Memory limit values. Only the CPU/Memory > requests values are configured. > For storage, we are using portworx. After deleting the wal directory, the > pod suddenly came up and started running. No restart for the last 21 hours. > However, I can expect the restart after 1 or 2 days. That is what's been > happening for more than 2 weeks. > > Thanks & Regards, > Anoop > > > On Thu, Oct 5, 2023 at 3:31 AM Ben Kochie <[email protected]> wrote: > >> *ts=2023-10-04T20:33:32.135Z caller=main.go:776 level=warn msg="Received >> SIGTERM, exiting gracefully..."* >> >> The time between your first log line and this line is about 15 minutes. >> Kubernetes is timing out the startup and terminating your process. >> >> Your startup is _very_ slow. Without any other information it's too hard >> to say why it's slow. Do you have CPU limits? What kind of storage? Too >> many variables. >> >> On Wed, Oct 4, 2023 at 10:52 PM Anoop <[email protected]> wrote: >> >>> Hi, >>> >>> I have configured Prometheus in Kubernetes with 2 replicas. One of the >>> two pods keeps restarting, but the other pod is working fine without any >>> issues. I could not find any errors when describing the pod. Resource >>> utilization is normal. The node has enough resources and storage available >>> to run the Prometheus pod and store the data. The Prometheus log for the >>> affected pod is provided below.. >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> *ts=2023-10-04T20:18:42.029Z caller=repair.go:57 level=info >>> component=tsdb msg="Found healthy block" mint=1696298400000 >>> maxt=1696305600000 >>> ulid=01HBSYGH69GH5KVHETGXCVA0K6ts=2023-10-04T20:18:42.030Z >>> caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a >>> previous execution already existed. It was replaced" >>> file=/prometheus/lockts=2023-10-04T20:18:43.722Z caller=head.go:488 >>> level=info component=tsdb msg="Replaying on-disk memory mappable chunks if >>> any"ts=2023-10-04T20:18:43.722Z caller=head.go:522 level=info >>> component=tsdb msg="On-disk memory mappable chunks replay completed" >>> duration=25.416µsts=2023-10-04T20:18:43.722Z caller=head.go:528 level=info >>> component=tsdb msg="Replaying WAL, this may take a >>> while"ts=2023-10-04T20:18:48.538Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=0 >>> maxSegment=97ts=2023-10-04T20:21:37.477Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=1 >>> maxSegment=97ts=2023-10-04T20:24:44.730Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=2 >>> maxSegment=97ts=2023-10-04T20:28:52.391Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=3 >>> maxSegment=97ts=2023-10-04T20:32:49.630Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=4 >>> maxSegment=97ts=2023-10-04T20:33:32.135Z caller=main.go:776 level=warn >>> msg="Received SIGTERM, exiting gracefully..."ts=2023-10-04T20:33:32.135Z >>> caller=main.go:799 level=info msg="Stopping scrape discovery >>> manager..."ts=2023-10-04T20:33:32.135Z caller=main.go:813 level=info >>> msg="Stopping notify discovery manager..."ts=2023-10-04T20:33:32.135Z >>> caller=main.go:835 level=info msg="Stopping scrape >>> manager..."ts=2023-10-04T20:33:32.135Z caller=main.go:829 level=info >>> msg="Scrape manager stopped"ts=2023-10-04T20:33:32.135Z caller=main.go:795 >>> level=info msg="Scrape discovery manager >>> stopped"ts=2023-10-04T20:33:32.135Z caller=main.go:809 level=info >>> msg="Notify discovery manager stopped"ts=2023-10-04T20:33:32.135Z >>> caller=manager.go:945 level=info component="rule manager" msg="Stopping >>> rule manager..."ts=2023-10-04T20:33:32.135Z caller=manager.go:955 >>> level=info component="rule manager" msg="Rule manager >>> stopped"ts=2023-10-04T20:33:32.135Z caller=notifier.go:600 level=info >>> component=notifier msg="Stopping notification >>> manager..."ts=2023-10-04T20:33:32.135Z caller=main.go:1055 level=info >>> msg="Notifier manager stopped"ts=2023-10-04T20:36:25.242Z >>> caller=head.go:599 level=info component=tsdb msg="WAL segment loaded" >>> segment=5 maxSegment=97ts=2023-10-04T20:37:16.354Z caller=head.go:599 >>> level=info component=tsdb msg="WAL segment loaded" segment=6 >>> maxSegment=97ts=2023-10-04T20:37:18.434Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=7 >>> maxSegment=97ts=2023-10-04T20:37:20.454Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=8 >>> maxSegment=97ts=2023-10-04T20:37:22.493Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=9 >>> maxSegment=97ts=2023-10-04T20:37:24.548Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=10 >>> maxSegment=97ts=2023-10-04T20:37:26.625Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=11 >>> maxSegment=97ts=2023-10-04T20:37:28.597Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=12 >>> maxSegment=97ts=2023-10-04T20:37:30.632Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=13 >>> maxSegment=97ts=2023-10-04T20:37:32.638Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=14 >>> maxSegment=97ts=2023-10-04T20:37:34.585Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=15 >>> maxSegment=97ts=2023-10-04T20:37:36.726Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=16 >>> maxSegment=97ts=2023-10-04T20:37:38.708Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=17 >>> maxSegment=97ts=2023-10-04T20:37:40.706Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=18 >>> maxSegment=97ts=2023-10-04T20:37:42.688Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=19 >>> maxSegment=97ts=2023-10-04T20:37:44.635Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=20 >>> maxSegment=97ts=2023-10-04T20:37:46.616Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=21 >>> maxSegment=97ts=2023-10-04T20:37:48.611Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=22 >>> maxSegment=97ts=2023-10-04T20:37:53.567Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=23 >>> maxSegment=97ts=2023-10-04T20:37:57.835Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=24 >>> maxSegment=97ts=2023-10-04T20:37:59.312Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=25 >>> maxSegment=97ts=2023-10-04T20:38:00.824Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=26 >>> maxSegment=97ts=2023-10-04T20:38:02.312Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=27 >>> maxSegment=97ts=2023-10-04T20:38:03.740Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=28 >>> maxSegment=97ts=2023-10-04T20:38:05.160Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=29 >>> maxSegment=97ts=2023-10-04T20:38:06.648Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=30 >>> maxSegment=97ts=2023-10-04T20:38:08.184Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=31 >>> maxSegment=97ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=32 >>> maxSegment=97ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=33 >>> maxSegment=97ts=2023-10-04T20:38:09.148Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=34 >>> maxSegment=97ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=35 >>> maxSegment=97ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=36 >>> maxSegment=97ts=2023-10-04T20:38:09.149Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=37 >>> maxSegment=97ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=38 >>> maxSegment=97ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=39 >>> maxSegment=97ts=2023-10-04T20:38:09.150Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=40 >>> maxSegment=97ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=41 >>> maxSegment=97ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=42 >>> maxSegment=97ts=2023-10-04T20:38:09.151Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=43 >>> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=44 >>> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=45 >>> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=46 >>> maxSegment=97ts=2023-10-04T20:38:09.152Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=47 >>> maxSegment=97ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=48 >>> maxSegment=97ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=49 >>> maxSegment=97ts=2023-10-04T20:38:09.153Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=50 >>> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=51 >>> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=52 >>> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=53 >>> maxSegment=97ts=2023-10-04T20:38:09.154Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=54 >>> maxSegment=97ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=55 >>> maxSegment=97ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=56 >>> maxSegment=97ts=2023-10-04T20:38:09.155Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=57 >>> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=58 >>> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=59 >>> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=60 >>> maxSegment=97ts=2023-10-04T20:38:09.156Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=61 >>> maxSegment=97ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=62 >>> maxSegment=97ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=63 >>> maxSegment=97ts=2023-10-04T20:38:09.157Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=64 >>> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=65 >>> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=66 >>> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=67 >>> maxSegment=97ts=2023-10-04T20:38:09.158Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=68 >>> maxSegment=97ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=69 >>> maxSegment=97ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=70 >>> maxSegment=97ts=2023-10-04T20:38:09.159Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=71 >>> maxSegment=97ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=72 >>> maxSegment=97ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=73 >>> maxSegment=97ts=2023-10-04T20:38:09.160Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=74 >>> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=75 >>> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=76 >>> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=77 >>> maxSegment=97ts=2023-10-04T20:38:09.161Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=78 >>> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=79 >>> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=80 >>> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=81 >>> maxSegment=97ts=2023-10-04T20:38:09.162Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=82 >>> maxSegment=97ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=83 >>> maxSegment=97ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=84 >>> maxSegment=97ts=2023-10-04T20:38:09.163Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=85 >>> maxSegment=97ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=86 >>> maxSegment=97ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=87 >>> maxSegment=97ts=2023-10-04T20:38:09.164Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=88 >>> maxSegment=97ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=89 >>> maxSegment=97ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=90 >>> maxSegment=97ts=2023-10-04T20:38:09.165Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=91 >>> maxSegment=97ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=92 >>> maxSegment=97ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=93 >>> maxSegment=97ts=2023-10-04T20:38:09.166Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=94 >>> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=95 >>> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=96 >>> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:599 level=info >>> component=tsdb msg="WAL segment loaded" segment=97 >>> maxSegment=97ts=2023-10-04T20:38:09.167Z caller=head.go:605 level=info >>> component=tsdb msg="WAL replay completed" >>> checkpoint_replay_duration=911.162µs wal_replay_duration=19m25.444199083s >>> total_replay_duration=19m25.445156166sts=2023-10-04T20:38:22.160Z >>> caller=main.go:945 level=info >>> fs_type=EXT4_SUPER_MAGICts=2023-10-04T20:38:22.160Z caller=main.go:948 >>> level=info msg="TSDB started"ts=2023-10-04T20:38:22.160Z >>> caller=main.go:1067 level=info msg="See you next time!"* >>> >>> Any thoughts? >>> >>> Thanks, >>> Anoop >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/fee699aa-d41d-4cd1-b70b-9adc8dc0db20n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/prometheus-users/fee699aa-d41d-4cd1-b70b-9adc8dc0db20n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/43bcc619-783b-4f9f-9cf6-19b1d39e28a5n%40googlegroups.com.
