Hi Vijayendra, what was the problem in the end? Sharing this knowledge might help other community member who might run into the same issue.
Cheers, Till On Wed, Sep 2, 2020 at 3:34 AM Vijayendra Yadav <contact....@gmail.com> wrote: > Thanks all, I could see the metrics. > > On Thu, Aug 27, 2020 at 7:51 AM Robert Metzger <rmetz...@apache.org> > wrote: > >> I don't think these error messages give us a hint why you can't see the >> metrics (because they are about registering metrics, not reporting them) >> >> Are you sure you are using the right configuration parameters for Flink >> 1.10? That all required JARs are in the lib/ folder (on all machines) and >> that your graphite setup is working (have you confirmed that you can show >> any metrics in the Graphite UI (maybe from a Graphite demo thingy))? >> >> >> On Thu, Aug 27, 2020 at 2:05 AM Vijayendra Yadav <contact....@gmail.com> >> wrote: >> >>> Hi Chesnay and Dawid, >>> >>> I see multiple entries as following in Log: >>> >>> 2020-08-26 23:46:19,105 WARN >>> org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while >>> registering metric: numRecordsIn. >>> java.lang.IllegalArgumentException: A metric named >>> ip-99-9999-99-99.taskmanager.container_1596056409708_1570_01_000006.vdcs-kafka-flink-test.Map.0.numRecordsIn >>> already exists >>> at >>> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >>> 2020-08-26 23:46:19,094 WARN >>> org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while >>> registering metric: numRecordsOut. >>> java.lang.IllegalArgumentException: A metric named >>> ip-99-9999-99-999.taskmanager.container_1596056409708_1570_01_000005.vdcs-kafka-flink-test.Map.2.numRecordsOut >>> already exists >>> at >>> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >>> at >>> org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) >>> at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >>> at >>> org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) >>> at >>> org.apache.flink.runtime.metrics.MetricRegistryImpl.register(MetricRegistryImpl.java:343) >>> at >>> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.addMetric(AbstractMetricGroup.java:426) >>> at >>> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:359) >>> at >>> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:349) >>> at >>> org.apache.flink.runtime.metrics.groups.OperatorIOMetricGroup.<init>(OperatorIOMetricGroup.java:41) >>> at >>> org.apache.flink.runtime.metrics.groups.OperatorMetricGroup.<init>(OperatorMetricGroup.java:48) >>> at >>> org.apache.flink.runtime.metrics.groups.TaskMetricGroup.lambda$getOrAddOperator$0(TaskMetricGroup.java:154) >>> at java.util.HashMap.computeIfAbsent(HashMap.java:1127) >>> at >>> org.apache.flink.runtime.metrics.groups.TaskMetricGroup.getOrAddOperator(TaskMetricGroup.java:154) >>> at >>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.setup(AbstractStreamOperator.java:180) >>> at >>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.setup(AbstractUdfStreamOperator.java:82) >>> at >>> org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:75) >>> at >>> org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:48) >>> at >>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:429) >>> at >>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:353) >>> at >>> org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:144) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:433) >>> at >>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461) >>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) >>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) >>> at java.lang.Thread.run(Thread.java:748) >>> Regards, >>> Vijay >>> >>> >>> On Wed, Aug 26, 2020 at 7:53 AM Chesnay Schepler <ches...@apache.org> >>> wrote: >>> >>>> metrics.reporter.grph.class: >>>> org.apache.flink.metrics.graphite.GraphiteReporter >>>> >>>> >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphitegraphitereporter >>>> >>>> On 26/08/2020 16:40, Vijayendra Yadav wrote: >>>> >>>> Hi Dawid, >>>> >>>> I have 1.10.0 version of flink. What is alternative for this version ? >>>> >>>> Regards, >>>> Vijay >>>> >>>> >>>> On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz <dwysakow...@apache.org> >>>> <dwysakow...@apache.org> wrote: >>>> >>>> >>>> >>>> Hi Vijay, >>>> >>>> I think the problem might be that you are using a wrong version of the >>>> reporter. >>>> >>>> You say you used flink-metrics-graphite-1.10.0.jar from 1.10 as a >>>> plugin, but it was migrated to plugins in 1.11 only[1]. >>>> >>>> I'd recommend trying it out with the same 1.11 version of Flink and >>>> Graphite reporter. >>>> >>>> Best, >>>> >>>> Dawid >>>> >>>> [1] https://issues.apache.org/jira/browse/FLINK-16965 >>>> On 26/08/2020 08:04, Vijayendra Yadav wrote: >>>> >>>> Hi Nikola, >>>> >>>> To rule out any other cluster issues, I have tried it in my local now. >>>> Steps as follows, but don't see any metrics yet. >>>> >>>> 1) Set up local Graphite >>>> >>>> docker run -d\ >>>> --name graphite\ >>>> --restart=always\ >>>> -p 80:80\ >>>> -p 2003-2004:2003-2004\ >>>> -p 2023-2024:2023-2024\ >>>> -p 8125:8125/udp\ >>>> -p 8126:8126\ >>>> graphiteapp/graphite-statsd >>>> >>>> Mapped Ports >>>> Host Container Service >>>> 80 80 nginx <https://www.nginx.com/resources/admin-guide/> >>>> 2003 2003 carbon receiver - plaintext >>>> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol> >>>> 2004 2004 carbon receiver - pickle >>>> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol> >>>> 2023 2023 carbon aggregator - plaintext >>>> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >>>> 2024 2024 carbon aggregator - pickle >>>> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >>>> 8080 8080 Graphite internal gunicorn port (without Nginx proxying). >>>> 8125 8125 statsd >>>> <https://github.com/etsy/statsd/blob/master/docs/server.md> >>>> 8126 8126 statsd admin >>>> <https://github.com/etsy/statsd/blob/master/docs/admin_interface.md> >>>> 2) WebUI: >>>> >>>> <image.png> >>>> >>>> >>>> >>>> 3) Run Flink example Job. >>>> ./bin/flink run >>>> ./examples/flink-examples-streaming_2.11-1.11-SNAPSHOT-SocketWindowWordCount.jar >>>> --port 9999 >>>> >>>> with conf/flink-conf.yaml set as: >>>> >>>> metrics.reporter.grph.factory.class: >>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>>> metrics.reporter.grph.host: localhost >>>> metrics.reporter.grph.port: 2003 >>>> metrics.reporter.grph.protocol: TCP >>>> metrics.reporter.grph.interval: 1 SECONDS >>>> >>>> and graphite jar: >>>> >>>> plugins/flink-metrics-graphite/flink-metrics-graphite-1.10.0.jar >>>> >>>> >>>> 4) Can't see any activity in webui graphite. >>>> >>>> >>>> Could you review and let me know what is wrong here ? any other way you >>>> suggest to be able to view the raw metrics data ? >>>> Also, do you have sample metrics raw format, you can share from any >>>> other project. >>>> >>>> Regards, >>>> Vijay >>>> >>>> >>>> >>>> >>>> On Sun, Aug 23, 2020 at 9:26 PM Nikola Hrusov <n.hru...@gmail.com> >>>> wrote: >>>> >>>>> Hi Vijay, >>>>> >>>>> Your steps look correct to me. >>>>> Perhaps you can double check that the graphite port you are sending is >>>>> correct? THe default carbon port is 2003 and if you use the aggregator it >>>>> is 2023. >>>>> >>>>> You should be able to see in both flink jobmanager and taskmanager >>>>> that the metrics have been initialized with the config you have pasted. >>>>> >>>>> Regards >>>>> , >>>>> Nikola Hrusov >>>>> >>>>> >>>>> On Mon, Aug 24, 2020 at 5:00 AM Vijayendra Yadav < >>>>> contact....@gmail.com> wrote: >>>>> >>>>>> Hi Team, >>>>>> >>>>>> I am trying to export Flink stream default metrics using Graphite, >>>>>> but I can't find it in the Graphite metrics console. Could you confirm >>>>>> the >>>>>> steps below are correct? >>>>>> >>>>>> *1) Updated flink-conf.yaml* >>>>>> >>>>>> metrics.reporter.grph.factory.class: >>>>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>>>>> metrics.reporter.grph.host: port >>>>>> metrics.reporter.grph.port: 9109 >>>>>> metrics.reporter.grph.protocol: TCP >>>>>> metrics.reporter.grph.interval: 30 SECONDS >>>>>> >>>>>> 2) Added Graphite jar in plugin folder : >>>>>> >>>>>> ll */usr/lib/flink/plugins/metric/* >>>>>> *flink-metrics-graphite-1.10.0.jar* >>>>>> >>>>>> 3) Looking metrics in graphite server: >>>>>> >>>>>> http://port:8080/metrics <http://10.108.58.63:8080/metrics> >>>>>> >>>>>> Note: No code change is done. >>>>>> >>>>>> Regards, >>>>>> Vijay >>>>>> >>>>>> >>>>>> >>>>
