Nice, thank you! Regards buggy client side timestamps <https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus#buggy-client-side-timestamps>, I didn't realise until now that "honor_timestamps: true" was the default. So for the OP, I should have said to try setting "honor_timestamps: false" instead of removing it. But in any case, snmp_exporter doesn't export timestamps <https://github.com/prometheus/snmp_exporter/discussions/815>, so it shouldn't make any difference in this case.
On Thursday, 15 December 2022 at 13:01:25 UTC [email protected] wrote: > Btw., even though we haven't found your specific problem yet, this thread > motivated me to write a blog post about these errors: > https://promlabs.com/blog/2022/12/15/understanding-duplicate-samples-and-out-of-order-timestamp-errors-in-prometheus > > On Tue, Dec 13, 2022 at 9:59 PM Umut Cokbilir <[email protected]> wrote: > >> Hi All, >> >> I tried commenting it out but it didn't solve the problem that I faced. >> I'll try 2.37.5 LTS version, if it solves then i'll update you. >> Thanks >> >> 13 Aralık 2022 Salı tarihinde saat 18:42:40 UTC+3 itibarıyla Brian >> Candler şunları yazdı: >> >>> Thank you for the full files (although you probably should have scrubbed >>> your snmp community) >>> >>> The only slightly suspicious thing I see is >>> honor_timestamps: true >>> which generally shouldn't be needed except in very special >>> circumstances. I don't *think* it will make any difference here, since >>> snmp_exporter doesn't give timestamps with its metrics anyway, but can you >>> try commenting it out? >>> >>> Otherwise, follow Julien's advice of trying the same configuration with >>> v2.37.5 (which is an LTS release) and see if the problem recurs. >>> >>> On Tuesday, 13 December 2022 at 15:28:37 UTC [email protected] wrote: >>> >>>> Hi All, >>>> >>>> I added my all config files attached. >>>> Thank you so much for this deep diving. >>>> >>>> 13 Aralık 2022 Salı tarihinde saat 17:37:43 UTC+3 itibarıyla Brian >>>> Candler şunları yazdı: >>>> >>>>> I ran your curl output through "promtool check metrics", but it only >>>>> warns about label and metric naming conventions (which SNMP violates to >>>>> be >>>>> consistent with standard MIB usage). There are no duplicate timeseries. >>>>> >>>>> # /opt/prometheus/promtool check metrics <ert3.txt 2>&1 | sort | uniq >>>>> -c >>>>> 11247 ifAlias label names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> 1 ifAlias metric names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> 13767 ifDescr label names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> 1 ifDescr metric names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> 1 ifHCInOctets counter metrics should have "_total" suffix >>>>> 11247 ifHCInOctets label names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> 1 ifHCInOctets metric names should be written in 'snake_case' >>>>> not 'camelCase' >>>>> 1 ifHCOutOctets counter metrics should have "_total" suffix >>>>> 11247 ifHCOutOctets label names should be written in 'snake_case' >>>>> not 'camelCase' >>>>> 1 ifHCOutOctets metric names should be written in 'snake_case' >>>>> not 'camelCase' >>>>> 1 sysName label names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> 1 sysName metric names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> 1 sysUpTime metric names should be written in 'snake_case' not >>>>> 'camelCase' >>>>> >>>>> Changing the target.yml file periodically won't make a difference; >>>>> Prometheus will just pick up the new one for the next scrape. So this is >>>>> odd. >>>>> >>>>> On Tuesday, 13 December 2022 at 12:21:15 UTC [email protected] >>>>> wrote: >>>>> >>>>>> Thanks for your detail explanation. You can see my answers below. >>>>>> >>>>>> I am not doing federation or any sort of availability. >>>>>> I've checked [1h] then [1d] and haven't seen any multiple data points >>>>>> :( >>>>>> I am not scraping the same targets twice but i am updating target.yml >>>>>> every hour. Does it affect? >>>>>> i am not sure there is any bug in my exporter config but it scrapes >>>>>> data from mydevices every 5 minutes. >>>>>> Please find curl output attached. >>>>>> >>>>>> >>>>>> 13 Aralık 2022 Salı tarihinde saat 14:44:31 UTC+3 itibarıyla Brian >>>>>> Candler şunları yazdı: >>>>>> >>>>>>> The first error explicitly mentions a particular metric >>>>>>> "fix:min_in", and the input to the recording rule which generates that >>>>>>> is >>>>>>> the metric "access:ifHCInOctets:FIX_rate" (rule 16, counting from zero) >>>>>>> But it also mentions an error for the metric >>>>>>> "access:ifHCInOctets:FIX_rate", which takes its input from >>>>>>> "ifHCInOctets" >>>>>>> (rule 4) >>>>>>> >>>>>>> So my guess is that you have duplicate data somewhere, posisbly in >>>>>>> ifHCInOctets. Try these queries: >>>>>>> >>>>>>> >>>>>>> ifHCInOctets{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h] >>>>>>> >>>>>>> access:ifHCInOctets:FIX_rate{instance="10.85.4.3",ifAlias="FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||"}[1h] >>>>>>> >>>>>>> Do you see multiple data points with the same timestamp? In which >>>>>>> timeseries? if not, try extending the time range to [1d] >>>>>>> >>>>>>> This would however require you to have multiple values of >>>>>>> ifHCInOctets with the same value of all labels (same ifAlias, same >>>>>>> ifDescr, >>>>>>> same ifIndex). >>>>>>> >>>>>>> If that's what you're seeing then you have to work backwards: are >>>>>>> you scraping the same target twice? Or is there a bug in your >>>>>>> snmp_exporter config? Try scraping the device manually using curl: >>>>>>> >>>>>>> curl -g 'localhost:9115/snmp?module=XXXX&target=10.85.4.3' >>>>>>> >>>>>>> (Having said that, if there were duplicates at this point, I'd be >>>>>>> expecting them to be rejected at scrape time). >>>>>>> >>>>>>> Are you doing any sort of high-availability or federation? If so, >>>>>>> you need different external labels for each data source. >>>>>>> >>>>>>> On Tuesday, 13 December 2022 at 10:55:19 UTC [email protected] >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Julius, >>>>>>>> >>>>>>>> Please find my recording rules file attached. There isn't any rules >>>>>>>> using same names. >>>>>>>> Please also check it? Where i am doing wrong? >>>>>>>> >>>>>>>> Thanks >>>>>>>> >>>>>>>> 13 Aralık 2022 Salı tarihinde saat 12:07:31 UTC+3 itibarıyla >>>>>>>> [email protected] şunları yazdı: >>>>>>>> >>>>>>>>> Hi Umut, >>>>>>>>> >>>>>>>>> This error occurs when something in Prometheus tries to append a >>>>>>>>> sample to a time series with the same timestamp as the last already >>>>>>>>> stored >>>>>>>>> sample in the series, but with a different sample value. Usually this >>>>>>>>> is >>>>>>>>> because of some accidental collision between time series identities >>>>>>>>> (metric >>>>>>>>> name + label set). Is there any chance that anything else in your >>>>>>>>> Prometheus setup also produces metrics with the same names as the >>>>>>>>> ones in >>>>>>>>> your recording rules? Maybe the recording rules are duplicated, but >>>>>>>>> with >>>>>>>>> different values (probably in the same rule group, since it's the >>>>>>>>> same >>>>>>>>> timestamp)? >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Julius >>>>>>>>> >>>>>>>>> On Tue, Dec 13, 2022 at 6:54 AM Umut Cokbilir <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> I couldn't understand the root cause of the problem below. How to >>>>>>>>>> solve it? >>>>>>>>>> >>>>>>>>>> Prometheus version: prometheus-2.40.3.linux-amd64 >>>>>>>>>> snmp_exporter version: snmp_exporter-0.20.0 >>>>>>>>>> >>>>>>>>>> Rules: >>>>>>>>>> - record: fix:min_in >>>>>>>>>> expr: min_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>>>> - record: fix:min_out >>>>>>>>>> expr: min_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>>>> - record: fix:max_in >>>>>>>>>> expr: max_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>>>> - record: fix:max_out >>>>>>>>>> expr: max_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>>>> - record: fix:avg_in >>>>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[1d]) >>>>>>>>>> - record: fix:avg_out >>>>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[1d]) >>>>>>>>>> - record: fix:avg_in:12h >>>>>>>>>> expr: avg_over_time(access:ifHCInOctets:FIX_rate[12h]) >>>>>>>>>> - record: fix:avg_out:12h >>>>>>>>>> expr: avg_over_time(access:ifHCOutOctets:FIX_rate[12h]) >>>>>>>>>> >>>>>>>>>> 04:41:23 --ts=2022-12-13T01:41:23.771Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:min_in >>>>>>>>>> index=16 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:min_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 22676.373333333333 @[1670895366553]" >>>>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.099Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 45519.09881820317 @[1670895366553]" >>>>>>>>>> 04:41:25 --ts=2022-12-13T01:41:25.495Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out >>>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 571800.1416907905 @[1670895366553]" >>>>>>>>>> 04:41:26 --ts=2022-12-13T01:41:25.864Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_in:12h >>>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 37126.089073190036 @[1670895366553]" >>>>>>>>>> 04:41:26 --ts=2022-12-13T01:41:26.208Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out:12h >>>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 568421.9923866978 @[1670895366553]" >>>>>>>>>> 05:21:23 --ts=2022-12-13T02:21:23.715Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 45353.12701316737 @[1670897766553]" >>>>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.126Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out >>>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 571748.0096685543 @[1670897766553]" >>>>>>>>>> 05:21:24 --ts=2022-12-13T02:21:24.453Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_in:12h >>>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 31094.769935320466 @[1670897766553]" >>>>>>>>>> 05:21:25 --ts=2022-12-13T02:21:24.794Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out:12h >>>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 567518.4603637857 @[1670897766553]" >>>>>>>>>> 06:06:26 --ts=2022-12-13T03:06:25.984Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 45189.166572681956 @[1670900466553]" >>>>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.353Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out >>>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 571701.6636125795 @[1670900466553]" >>>>>>>>>> 06:06:26 --ts=2022-12-13T03:06:26.679Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_in:12h >>>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 29880.13207330078 @[1670900466553]" >>>>>>>>>> 06:06:27 --ts=2022-12-13T03:06:27.010Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out:12h >>>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 567379.9435644317 @[1670900466553]" >>>>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.089Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 45110.958791205536 @[1670901366553]" >>>>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.496Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out >>>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 571723.652811564 @[1670901366553]" >>>>>>>>>> 06:21:27 --ts=2022-12-13T03:21:27.832Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_in:12h >>>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 29745.59755724392 @[1670901366553]" >>>>>>>>>> 06:21:28 --ts=2022-12-13T03:21:28.173Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out:12h >>>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 567638.6622438562 @[1670901366553]" >>>>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.058Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 44958.979497887565 @[1670903466553]" >>>>>>>>>> 06:56:28 --ts=2022-12-13T03:56:28.650Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out >>>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 571641.2677371124 @[1670903466553]" >>>>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.089Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_in:12h >>>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 29316.18369902085 @[1670903466553]" >>>>>>>>>> 06:56:29 --ts=2022-12-13T03:56:29.713Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out:12h >>>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 567614.2432218753 @[1670903466553]" >>>>>>>>>> 07:06:08 --ts=2022-12-13T04:06:07.854Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 26498.213333333337 @[1670904066553]" >>>>>>>>>> 07:06:10 --ts=2022-12-13T04:06:09.708Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 569222.2666666667 @[1670904066553]" >>>>>>>>>> 07:06:12 --ts=2022-12-13T04:06:11.181Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation >>>>>>>>>> result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 26498.213333333337 @[1670904066553]" >>>>>>>>>> 07:06:13 --ts=2022-12-13T04:06:11.676Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation >>>>>>>>>> result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 569222.2666666667 @[1670904066553]" >>>>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.065Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 44893.41348050907 @[1670906766553]" >>>>>>>>>> 07:51:27 --ts=2022-12-13T04:51:27.564Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out >>>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 571623.9997716311 @[1670906766553]" >>>>>>>>>> 07:51:28 --ts=2022-12-13T04:51:27.907Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_in:12h >>>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 28273.031953867194 @[1670906766553]" >>>>>>>>>> 07:51:28 --ts=2022-12-13T04:51:28.245Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out:12h >>>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 568057.166386401 @[1670906766553]" >>>>>>>>>> 08:01:08 --ts=2022-12-13T05:01:08.140Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCInOctets:rate index=0 msg="Rule evaluation result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCInOctets:rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 26605.573333333334 @[1670907366553]" >>>>>>>>>> 08:01:10 --ts=2022-12-13T05:01:09.841Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCOutOctets:rate index=1 msg="Rule evaluation result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 566597.4400000001 @[1670907366553]" >>>>>>>>>> 08:01:12 --ts=2022-12-13T05:01:11.039Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCInOctets:FIX_rate index=4 msg="Rule evaluation >>>>>>>>>> result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCInOctets:FIX_rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 26605.573333333334 @[1670907366553]" >>>>>>>>>> 08:01:13 --ts=2022-12-13T05:01:11.295Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=access:ifHCOutOctets:FIX_rate index=5 msg="Rule evaluation >>>>>>>>>> result >>>>>>>>>> discarded" err="duplicate sample for timestamp" >>>>>>>>>> sample="{__name__=\"access:ifHCOutOctets:FIX_rate\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 566597.4400000001 @[1670907366553]" >>>>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.233Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw name=fix:avg_in >>>>>>>>>> index=20 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 44828.267673529954 @[1670908266553]" >>>>>>>>>> 08:16:26 --ts=2022-12-13T05:16:26.661Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out >>>>>>>>>> index=21 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 571600.3475860697 @[1670908266553]" >>>>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.003Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_in:12h >>>>>>>>>> index=22 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_in:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 27422.084942333982 @[1670908266553]" >>>>>>>>>> 08:16:27 --ts=2022-12-13T05:16:27.338Z caller=manager.go:684 >>>>>>>>>> level=warn component="rule manager" >>>>>>>>>> file=/Products/monitoring_tool/rules.yml group=accnw >>>>>>>>>> name=fix:avg_out:12h >>>>>>>>>> index=23 msg="Rule evaluation result discarded" err="duplicate >>>>>>>>>> sample for >>>>>>>>>> timestamp" sample="{__name__=\"fix:avg_out:12h\", >>>>>>>>>> ifAlias=\"FIX_abb_ortakoy_mezarlik_l3vpn.vfrl-@26,330@-||10||\", >>>>>>>>>> ifDescr=\"Eth-Trunk253.3155\", ifIndex=\"9578\", >>>>>>>>>> instance=\"10.85.4.3\", >>>>>>>>>> job=\"mtx\", sysName=\"1673(3)-PTN3120592_PURSAKLAR_PLAZA\"} => >>>>>>>>>> 568112.2042330011 @[1670908266553]" >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "Prometheus Users" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/prometheus-users/0f1c9697-8c58-4895-8917-182a66ed217dn%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Julius Volz >>>>>>>>> PromLabs - promlabs.com >>>>>>>>> >>>>>>>> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/05123d4b-11de-41b8-bbf0-d8332b6b76ebn%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/05123d4b-11de-41b8-bbf0-d8332b6b76ebn%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > > > -- > Julius Volz > PromLabs - promlabs.com > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3fad920f-6f78-496c-957c-58f8cc216a33n%40googlegroups.com.
