This sounds like you might have run into the Go timer jitter bug. Try enabling timestamp tolerance to mitigate the effect of timer jitter: https://promlabs.com/blog/2021/09/14/whats-new-in-prometheus-2-30/#improving-storage-efficiency-by-tuning-timestamp-tolerances
I have so far not found a satisfactory resolution to this bug, and even though enabling timestamp jitter tolerance helps, I still occasionally see instances inexplicably using up to 2.5x their previous average bytes per sample after a restart. It seems to be a lucky dip. Restarting the instance usually settles back down to the original and expected bytes per sample. On Wednesday, April 12, 2023 at 9:55:10 AM UTC+2 Per Carlson wrote: > Hi. > > We have a pair of Prometheus instances that consume significantly > different amounts of storage. The instances are created by the same > StatefulSet (created by Prometheus-operator), so they are using the same > configuration. > > Both instances have similar number of samples and series, but instance > "0" consume up to ~50% more storage than instance "1". > > $ kubectl exec prometheus-prometheus-0 -- /bin/sh -c "promtool tsdb list ." > BLOCK ULID MIN TIME MAX TIME DURATION > NUM SAMPLES NUM CHUNKS NUM SERIES SIZE > 01GWY0R4N3QG1QJS957XZ0SYP7 1680264000000 1680328800000 18h0m0s > 3296299059 26900037 931315 7259935610 > 01GX05E79R6WNQ4F6MMB068WJ7 1680328800003 1680393600000 17h59m59.997s > 3312300492 27012299 892364 7265602468 > 01GX237SWZYZ6X5XXMENGMQ1YM 1680393600002 1680458400000 17h59m59.998s > 3315540127 27036907 894595 7247593445 > 01GX410BDAPBMPKP100300C7DW 1680458400001 1680523200000 17h59m59.999s > 3320458065 27130364 987454 7328750825 > 01GX5YTZVD5W97D497JA11CATT 1680523200007 1680588000000 17h59m59.993s > 3318443269 27135815 1007206 7380926789 > 01GX7WMF1FNJ6MGT0TJY2A5KEM 1680588000001 1680652800000 17h59m59.999s > 3331999517 27259726 1028363 7364976990 > 01GX9TDYMKVWJ9WYYY7CD8BCWH 1680652800005 1680717600000 17h59m59.995s > 3327868238 27186293 981912 7288127305 > 01GXBR7FYPSRWA6N6313MMR9BM 1680717600009 1680782400000 17h59m59.991s > 3327937718 27125975 896286 7199443835 > 01GXDP01QKKXC137B7JJZ6706W 1680782400001 1680847200000 17h59m59.999s > 3333037262 27172805 897459 7194002011 > 01GXFKTGVZN6RXRM74PXB5E61Q 1680847200001 1680912000000 17h59m59.999s > 3329211104 27134065 879001 7202044230 > 01GXHHM1JST118SYQNX8Z5W8PX 1680912000004 1680976800000 17h59m59.996s > 3329464442 27131788 876881 7192136400 > 01GXKFCM51YQFDAWTGP6BBXGQZ 1680976800003 1681041600000 17h59m59.997s > 3329134675 27127804 875877 7197030123 > 01GXND71ZF62MK8M1DP5E7345M 1681041600011 1681106400000 17h59m59.989s > 3327555787 27119184 887763 7216837469 > 01GXQB0QJX2T55FBFFSZC9PC4D 1681106400005 1681171200000 17h59m59.995s > 3324035858 27084455 871653 7195109123 > 01GXS8T0EJXHH2C1B0CBAEPHQB 1681171200011 1681236000000 17h59m59.989s > 3315573555 26493111 989655 6235040678 > 01GXSXCRNTNDJC359R7160ZEDX 1681236000001 1681257600000 5h59m59.999s > 1107306526 9028997 828578 1830084344 > 01GXSPFKRCVXHRSD2WAFEFM0WD 1681257600005 1681264800000 1h59m59.995s > 369706839 3015597 808854 671597409 > 01GXSXBRED7JT7WJY9318QYKKZ 1681264800002 1681272000000 1h59m59.998s > 369661386 3012000 805553 668951473 > 01GXT47FQYC712E0M6XPSP41FF 1681272000001 1681279200000 1h59m59.999s > 369740628 3021714 823966 673649781 > > $ kubectl exec prometheus-prometheus-1 -- /bin/sh -c "promtool tsdb list ." > BLOCK ULID MIN TIME MAX TIME DURATION > NUM SAMPLES NUM CHUNKS NUM SERIES SIZE > 01GWY0RDK93D2RYJBHJRDMS100 1680264000000 1680328800000 18h0m0s > 3296396516 26926127 957040 4831014683 > 01GX05ETVBDXMQH0KW9NX7RCPC 1680328800003 1680393600000 17h59m59.997s > 3312324642 27036260 917296 4807892522 > 01GX2383F7YPDX400MN4DQ9CSX 1680393600002 1680458400000 17h59m59.998s > 3315587751 27059963 918166 4832761551 > 01GX410PJXX52PKFVH1H205385 1680458400003 1680523200000 17h59m59.997s > 3320397897 27157090 1014022 4890962085 > 01GX5YVKEAQ6D1NZM1AQW0YJ90 1680523200003 1680588000000 17h59m59.997s > 3318472581 27171422 1042831 4854062752 > 01GX7WMWV41M3PW3BFV62P0M32 1680588000001 1680652800000 17h59m59.999s > 3331918609 27288755 1056267 4861196239 > 01GX9TECS126QJM1A1F61GW0ZT 1680652800003 1680717600000 17h59m59.997s > 3328065112 27214643 1008335 4831609465 > 01GXBR7NZ3RXVSP50V5J2QE4HQ 1680717600003 1680782400000 17h59m59.997s > 3327954927 27159515 929150 4800273178 > 01GXDP1BCZF7THHQTSK4YGAYX7 1680782400001 1680847200000 17h59m59.999s > 3333181469 27202041 926662 4805982300 > 01GXFKTTFMZ723VQ94BDY1H1T9 1680847200001 1680912000000 17h59m59.999s > 3329426122 27163849 908757 4715850967 > 01GXHHMTBPDXX173MD9V1B24E7 1680912000003 1680976800000 17h59m59.997s > 3329413976 27163579 909408 4791199693 > 01GXKFCZX5HX3KBR82Y5H8DS3D 1680976800003 1681041600000 17h59m59.997s > 3329270278 27160688 908615 4789400558 > 01GXND7FRJ628BCK936DYH6BVV 1681041600003 1681106400000 17h59m59.997s > 3327613909 27155028 922458 4784598143 > 01GXQB1BWNS0EDT4TX7HFACWBB 1681106400003 1681171200000 17h59m59.997s > 3324104564 27121755 909230 4777589596 > 01GXS8TD8CA5E4BPWKY1VXEV71 1681171200003 1681236000000 17h59m59.997s > 3320030145 26512567 1016662 4802788228 > 01GXSXCQE71EB940GMBPS4NVXV 1681236000001 1681257600000 5h59m59.999s > 1107449702 9035526 835180 1609217929 > 01GXSPFXTNAJCYMSKABCF18YH9 1681257600005 1681264800000 1h59m59.995s > 369710007 3018579 811751 607549046 > 01GXSXBN258J5VHMXR1FG9E84P 1681264800002 1681272000000 1h59m59.998s > 369666717 3014789 808149 601091436 > 01GXT47CA6RV2QPZMVN45EH6CQ 1681272000000 1681279200000 2h0m0s > 369734916 3022662 825536 607677006 > > What can cause this type of effects? As both instances scrape the same > targets, we should achieve the same level of compression of the time series > data (AFAICT). > > -- > Pelle > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/523e9e62-76aa-4f21-80a4-e17462e8fc8en%40googlegroups.com.
