I don't think these error messages give us a hint why you can't see the metrics (because they are about registering metrics, not reporting them)
Are you sure you are using the right configuration parameters for Flink 1.10? That all required JARs are in the lib/ folder (on all machines) and that your graphite setup is working (have you confirmed that you can show any metrics in the Graphite UI (maybe from a Graphite demo thingy))? On Thu, Aug 27, 2020 at 2:05 AM Vijayendra Yadav <contact....@gmail.com> wrote: > Hi Chesnay and Dawid, > > I see multiple entries as following in Log: > > 2020-08-26 23:46:19,105 WARN > org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while > registering metric: numRecordsIn. > java.lang.IllegalArgumentException: A metric named > ip-99-9999-99-99.taskmanager.container_1596056409708_1570_01_000006.vdcs-kafka-flink-test.Map.0.numRecordsIn > already exists > at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) > 2020-08-26 23:46:19,094 WARN > org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while > registering metric: numRecordsOut. > java.lang.IllegalArgumentException: A metric named > ip-99-9999-99-999.taskmanager.container_1596056409708_1570_01_000005.vdcs-kafka-flink-test.Map.2.numRecordsOut > already exists > at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) > at > org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) > at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) > at > org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) > at > org.apache.flink.runtime.metrics.MetricRegistryImpl.register(MetricRegistryImpl.java:343) > at > org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.addMetric(AbstractMetricGroup.java:426) > at > org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:359) > at > org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:349) > at > org.apache.flink.runtime.metrics.groups.OperatorIOMetricGroup.<init>(OperatorIOMetricGroup.java:41) > at > org.apache.flink.runtime.metrics.groups.OperatorMetricGroup.<init>(OperatorMetricGroup.java:48) > at > org.apache.flink.runtime.metrics.groups.TaskMetricGroup.lambda$getOrAddOperator$0(TaskMetricGroup.java:154) > at java.util.HashMap.computeIfAbsent(HashMap.java:1127) > at > org.apache.flink.runtime.metrics.groups.TaskMetricGroup.getOrAddOperator(TaskMetricGroup.java:154) > at > org.apache.flink.streaming.api.operators.AbstractStreamOperator.setup(AbstractStreamOperator.java:180) > at > org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.setup(AbstractUdfStreamOperator.java:82) > at > org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:75) > at > org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:48) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:429) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:353) > at > org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:144) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:433) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) > at java.lang.Thread.run(Thread.java:748) > Regards, > Vijay > > > On Wed, Aug 26, 2020 at 7:53 AM Chesnay Schepler <ches...@apache.org> > wrote: > >> metrics.reporter.grph.class: >> org.apache.flink.metrics.graphite.GraphiteReporter >> >> >> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphitegraphitereporter >> >> On 26/08/2020 16:40, Vijayendra Yadav wrote: >> >> Hi Dawid, >> >> I have 1.10.0 version of flink. What is alternative for this version ? >> >> Regards, >> Vijay >> >> >> On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz <dwysakow...@apache.org> >> <dwysakow...@apache.org> wrote: >> >> >> >> Hi Vijay, >> >> I think the problem might be that you are using a wrong version of the >> reporter. >> >> You say you used flink-metrics-graphite-1.10.0.jar from 1.10 as a plugin, >> but it was migrated to plugins in 1.11 only[1]. >> >> I'd recommend trying it out with the same 1.11 version of Flink and >> Graphite reporter. >> >> Best, >> >> Dawid >> >> [1] https://issues.apache.org/jira/browse/FLINK-16965 >> On 26/08/2020 08:04, Vijayendra Yadav wrote: >> >> Hi Nikola, >> >> To rule out any other cluster issues, I have tried it in my local now. >> Steps as follows, but don't see any metrics yet. >> >> 1) Set up local Graphite >> >> docker run -d\ >> --name graphite\ >> --restart=always\ >> -p 80:80\ >> -p 2003-2004:2003-2004\ >> -p 2023-2024:2023-2024\ >> -p 8125:8125/udp\ >> -p 8126:8126\ >> graphiteapp/graphite-statsd >> >> Mapped Ports >> Host Container Service >> 80 80 nginx <https://www.nginx.com/resources/admin-guide/> >> 2003 2003 carbon receiver - plaintext >> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol> >> 2004 2004 carbon receiver - pickle >> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol> >> 2023 2023 carbon aggregator - plaintext >> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >> 2024 2024 carbon aggregator - pickle >> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >> 8080 8080 Graphite internal gunicorn port (without Nginx proxying). >> 8125 8125 statsd >> <https://github.com/etsy/statsd/blob/master/docs/server.md> >> 8126 8126 statsd admin >> <https://github.com/etsy/statsd/blob/master/docs/admin_interface.md> >> 2) WebUI: >> >> <image.png> >> >> >> >> 3) Run Flink example Job. >> ./bin/flink run >> ./examples/flink-examples-streaming_2.11-1.11-SNAPSHOT-SocketWindowWordCount.jar >> --port 9999 >> >> with conf/flink-conf.yaml set as: >> >> metrics.reporter.grph.factory.class: >> org.apache.flink.metrics.graphite.GraphiteReporterFactory >> metrics.reporter.grph.host: localhost >> metrics.reporter.grph.port: 2003 >> metrics.reporter.grph.protocol: TCP >> metrics.reporter.grph.interval: 1 SECONDS >> >> and graphite jar: >> >> plugins/flink-metrics-graphite/flink-metrics-graphite-1.10.0.jar >> >> >> 4) Can't see any activity in webui graphite. >> >> >> Could you review and let me know what is wrong here ? any other way you >> suggest to be able to view the raw metrics data ? >> Also, do you have sample metrics raw format, you can share from any other >> project. >> >> Regards, >> Vijay >> >> >> >> >> On Sun, Aug 23, 2020 at 9:26 PM Nikola Hrusov <n.hru...@gmail.com> wrote: >> >>> Hi Vijay, >>> >>> Your steps look correct to me. >>> Perhaps you can double check that the graphite port you are sending is >>> correct? THe default carbon port is 2003 and if you use the aggregator it >>> is 2023. >>> >>> You should be able to see in both flink jobmanager and taskmanager that >>> the metrics have been initialized with the config you have pasted. >>> >>> Regards >>> , >>> Nikola Hrusov >>> >>> >>> On Mon, Aug 24, 2020 at 5:00 AM Vijayendra Yadav <contact....@gmail.com> >>> wrote: >>> >>>> Hi Team, >>>> >>>> I am trying to export Flink stream default metrics using Graphite, but >>>> I can't find it in the Graphite metrics console. Could you confirm the >>>> steps below are correct? >>>> >>>> *1) Updated flink-conf.yaml* >>>> >>>> metrics.reporter.grph.factory.class: >>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>>> metrics.reporter.grph.host: port >>>> metrics.reporter.grph.port: 9109 >>>> metrics.reporter.grph.protocol: TCP >>>> metrics.reporter.grph.interval: 30 SECONDS >>>> >>>> 2) Added Graphite jar in plugin folder : >>>> >>>> ll */usr/lib/flink/plugins/metric/* >>>> *flink-metrics-graphite-1.10.0.jar* >>>> >>>> 3) Looking metrics in graphite server: >>>> >>>> http://port:8080/metrics <http://10.108.58.63:8080/metrics> >>>> >>>> Note: No code change is done. >>>> >>>> Regards, >>>> Vijay >>>> >>>> >>>> >>
