Btw., even though we haven't found your specific problem yet, this thread motivated me to write a blog post about these errors: https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus
On Tue, Dec 13, 2022 at 9:59 PM Umut Cokbilir <[email protected]> wrote: > Hi All, > > I tried commenting it out but it didn't solve the problem that I faced. > I'll try 2.37.5 LTS version, if it solves then i'll update you. > Thanks > > 13 Aralık 2022 Salı tarihinde saat 18:42:40 UTC+3 itibarıyla Brian Candler > şunları yazdı: > >> Thank you for the full files (although you probably should have scrubbed >> your snmp community) >> >> The only slightly suspicious thing I see is >> honor_timestamps: true >> which generally shouldn't be needed except in very special >> circumstances. I don't *think* it will make any difference here, since >> snmp_exporter doesn't give timestamps with its metrics anyway, but can you >> try commenting it out? >> >> Otherwise, follow Julien's advice of trying the same configuration with >> v2.37.5 (which is an LTS release) and see if the problem recurs. >> >> On Tuesday, 13 December 2022 at 15:28:37 UTC [email protected] wrote: >> >>> Hi All, >>> >>> I added my all config files attached. >>> Thank you so much for this deep diving. >>> >>> 13 Aralık 2022 Salı tarihinde saat 17:37:43 UTC+3 itibarıyla Brian >>> Candler şunları yazdı: >>> >>>> I ran your curl output through "promtool check metrics", but it only >>>> warns about label and metric naming conventions (which SNMP violates to be >>>> consistent with standard MIB usage). There are no duplicate timeseries. >>>> >>>> # /opt/prometheus/promtool check metrics <ert3.txt 2>&1 | sort | uniq -c >>>> 11247 ifAlias label names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 ifAlias metric names should be written in 'snake_case' not >>>> 'camelCase' >>>> 13767 ifDescr label names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 ifDescr metric names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 ifHCInOctets counter metrics should have "_total" suffix >>>> 11247 ifHCInOctets label names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 ifHCInOctets metric names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 ifHCOutOctets counter metrics should have "_total" suffix >>>> 11247 ifHCOutOctets label names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 ifHCOutOctets metric names should be written in 'snake_case' >>>> not 'camelCase' >>>> 1 sysName label names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 sysName metric names should be written in 'snake_case' not >>>> 'camelCase' >>>> 1 sysUpTime metric names should be written in 'snake_case' not >>>> 'camelCase' >>>> >>>> Changing the target.yml file periodically won't make a difference; >>>> Prometheus will just pick up the new one for the next scrape. So this is >>>> odd. >>>> >>>> On Tuesday, 13 December 2022 at 12:21:15 UTC [email protected] wrote: >>>> >>>>> Thanks for your detail explanation. You can see my answers below. >>>>> >>>>> I am not doing federation or any sort of availability. >>>>> I've checked [1h] then [1d] and haven't seen any multiple data points >>>>> :( >>>>> I am not scraping the same targets twice but i am updating target.yml >>>>> every hour. Does it affect? >>>>> i am not sure there is any bug in my exporter config but it scrapes >>>>> data from mydevices every 5 minutes. >>>>> Please find curl output attached. >>>>> >>>>> >>>>> 13 Aralık 2022 Salı tarihinde saat 14:44:31 UTC+3 itibarıyla Brian >>>>> Candler şunları yazdı: >>>>> >>>>>> The first error explicitly mentions a particular metric "fix:min_in", >>>>>> and the input to the recording rule which generates that is the metric >>>>>> "access:ifHCInOctets:FIX_rate" (rule 16, counting from zero) >>>>>> But it also mentions an error for the metric >>>>>> "access:ifHCInOctets:FIX_rate", which takes its input from "ifHCInOctets" >>>>>> (rule 4) >>>>>> >>>>>> So my guess is that you have duplicate data somewhere, posisbly in >>>>>> ifHCInOctets. Try these queries: >>>>>> >>>>>> >>>>>> ifHCInOctets{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26 >>>>>> ,330@-||10||"}[1h] >>>>>> >>>>>> access:ifHCInOctets:FIX_rate{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26 >>>>>> ,330@-||10||"}[1h] >>>>>> >>>>>> Do you see multiple data points with the same timestamp? In which >>>>>> timeseries? if not, try extending the time range to [1d] >>>>>> >>>>>> This would however require you to have multiple values of >>>>>> ifHCInOctets with the same value of all labels (same ifAlias, same >>>>>> ifDescr, >>>>>> same ifIndex). >>>>>> >>>>>> If that's what you're seeing then you have to work backwards: are you >>>>>> scraping the same target twice? Or is there a bug in your snmp_exporter >>>>>> config? Try scraping the device manually using curl: >>>>>> >>>>>> curl -g 'localhost:9115/snmp?module=XXXX&target=10.85.4.3' >>>>>> >>>>>> (Having said that, if there were duplicates at this point, I'd be >>>>>> expecting them to be rejected at scrape time). >>>>>> >>>>>> Are you doing any sort of high-availability or federation? If so, you >>>>>> need different external labels for each data source. >>>>>> >>>>>> On Tuesday, 13 December 2022 at 10:55:19 UTC [email protected] >>>>>> wrote: >>>>>> >>>>>>> Hi Julius, >>>>>>> >>>>>>> Please find my recording rules file attached. There isn't any rules >>>>>>> using same names. >>>>>>> Please also check it? Where i am doing wrong? >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> 13 Aralık 2022 Salı tarihinde saat 12:07:31 UTC+3 itibarıyla >>>>>>> [email protected] şunları yazdı: >>>>>>> >>>>>>>> Hi Umut, >>>>>>>> >>>>>>>> This error occurs when something in Prometheus tries to append a >>>>>>>> sample to a time series with the same timestamp as the last already >>>>>>>> stored >>>>>>>> sample in the series, but with a different sample value. Usually this >>>>>>>> is >>>>>>>> because of some accidental collision between time series identities >>>>>>>> (metric >>>>>>>> name + label set). Is there any chance that anything else in your >>>>>>>> Prometheus setup also produces metrics with the same names as the ones >>>>>>>> in >>>>>>>> your recording rules? Maybe the recording rules are duplicated, but >>>>>>>> with >>>>>>>> different values (probably in the same rule group, since it's the same >>>>>>>> timestamp)? >>>>>>>> >>>>>>>> Regards, >>>>>>>> Julius >>>>>>>> >>>>>>>> On Tue, Dec 13, 2022 at 6:54 AM Umut Cokbilir <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I couldn't understand the root cause of the problem below. How to >>>>>>>>> solve it? >>>>>>>>> >>>>>>>>> Prometheus version: prometheus-2.40.3.linux-amd64 >>>>>>>>> snmp_exporter version: snmp_exporter-0.20.0 >>>>>>>>> >>>>>>>>> Rules: >>>>>>>>> - record: fix:min_in >>>>>>>>> expr: min_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>>> - record: fix:min_out >>>>>>>>> expr: min_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>>> - record: fix:max_in >>>>>>>>> expr: max_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>>> - record: fix:max_out >>>>>>>>> expr: max_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>>> - record: fix:avg_in >>>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>>> - record: fix:avg_out >>>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>>> - record: fix:avg_in:12h >>>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[12h]) >>>>>>>>> - record: fix:avg_out:12h >>>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[12h]) >>>>>>>>> >>>>>>>>> 04:41:23 --ts=2022-12-13T01:41:23.771Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:min_in >>>>>>>>> index=16 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:min_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 22676.373333333333 @[1670895366553]" >>>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.099Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 45519.09881820317 @[1670895366553]" >>>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.495Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 571800.1416907905 @[1670895366553]" >>>>>>>>> 04:41:26 --ts=2022-12-13T01:41:25.864Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_in:12h >>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 37126.089073190036 @[1670895366553]" >>>>>>>>> 04:41:26 --ts=2022-12-13T01:41:26.208Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_out:12h >>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 568421.9923866978 @[1670895366553]" >>>>>>>>> 05:21:23 --ts=2022-12-13T02:21:23.715Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 45353.12701316737 @[1670897766553]" >>>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.126Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 571748.0096685543 @[1670897766553]" >>>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.453Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_in:12h >>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 31094.769935320466 @[1670897766553]" >>>>>>>>> 05:21:25 --ts=2022-12-13T02:21:24.794Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_out:12h >>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 567518.4603637857 @[1670897766553]" >>>>>>>>> 06:06:26 --ts=2022-12-13T03:06:25.984Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 45189.166572681956 @[1670900466553]" >>>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.353Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 571701.6636125795 @[1670900466553]" >>>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.679Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_in:12h >>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 29880.13207330078 @[1670900466553]" >>>>>>>>> 06:06:27 --ts=2022-12-13T03:06:27.010Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_out:12h >>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 567379.9435644317 @[1670900466553]" >>>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.089Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 45110.958791205536 @[1670901366553]" >>>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.496Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 571723.652811564 @[1670901366553]" >>>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.832Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_in:12h >>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 29745.59755724392 @[1670901366553]" >>>>>>>>> 06:21:28 --ts=2022-12-13T03:21:28.173Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_out:12h >>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 567638.6622438562 @[1670901366553]" >>>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.058Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 44958.979497887565 @[1670903466553]" >>>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.650Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 571641.2677371124 @[1670903466553]" >>>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.089Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_in:12h >>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 29316.18369902085 @[1670903466553]" >>>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.713Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_out:12h >>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 567614.2432218753 @[1670903466553]" >>>>>>>>> 07:06:08 --ts=2022-12-13T04:06:07.854Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 26498.213333333337 @[1670904066553]" >>>>>>>>> 07:06:10 --ts=2022-12-13T04:06:09.708Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 569222.2666666667 @[1670904066553]" >>>>>>>>> 07:06:12 --ts=2022-12-13T04:06:11.181Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 26498.213333333337 @[1670904066553]" >>>>>>>>> 07:06:13 --ts=2022-12-13T04:06:11.676Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 569222.2666666667 @[1670904066553]" >>>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.065Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 44893.41348050907 @[1670906766553]" >>>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.564Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 571623.9997716311 @[1670906766553]" >>>>>>>>> 07:51:28 --ts=2022-12-13T04:51:27.907Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_in:12h >>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 28273.031953867194 @[1670906766553]" >>>>>>>>> 07:51:28 --ts=2022-12-13T04:51:28.245Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_out:12h >>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 568057.166386401 @[1670906766553]" >>>>>>>>> 08:01:08 --ts=2022-12-13T05:01:08.140Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 26605.573333333334 @[1670907366553]" >>>>>>>>> 08:01:10 --ts=2022-12-13T05:01:09.841Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 566597.4400000001 @[1670907366553]" >>>>>>>>> 08:01:12 --ts=2022-12-13T05:01:11.039Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 26605.573333333334 @[1670907366553]" >>>>>>>>> 08:01:13 --ts=2022-12-13T05:01:11.295Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation result >>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 566597.4400000001 @[1670907366553]" >>>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.233Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 44828.267673529954 @[1670908266553]" >>>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.661Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_out >>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 571600.3475860697 @[1670908266553]" >>>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.003Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_in:12h >>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 27422.084942333982 @[1670908266553]" >>>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.338Z caller=manager.go:684 >>>>>>>>> level=warn component="rule manager" >>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>> name=fix:avg_out:12h >>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate sample >>>>>>>>> for >>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>> 568112.2042330011 @[1670908266553]" >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Prometheus Users" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Julius Volz >>>>>>>> PromLabs - promlabs.com >>>>>>>> >>>>>>> -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/05123d4b-11de-41b8-bbf0-d8332b6b76ebn%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/05123d4b-11de-41b8-bbf0-d8332b6b76ebn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- Julius Volz PromLabs - promlabs.com -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAObpH5xoXo%3DzbnTwjfXo--JJ5joCwbCvaHezoWBswtDPyYuhPw%40mail.gmail.com.
