Thanks all, I could see the metrics. On Thu, Aug 27, 2020 at 7:51 AM Robert Metzger <rmetz...@apache.org> wrote:
> I don't think these error messages give us a hint why you can't see the > metrics (because they are about registering metrics, not reporting them) > > Are you sure you are using the right configuration parameters for Flink > 1.10? That all required JARs are in the lib/ folder (on all machines) and > that your graphite setup is working (have you confirmed that you can show > any metrics in the Graphite UI (maybe from a Graphite demo thingy))? > > > On Thu, Aug 27, 2020 at 2:05 AM Vijayendra Yadav <contact....@gmail.com> > wrote: > >> Hi Chesnay and Dawid, >> >> I see multiple entries as following in Log: >> >> 2020-08-26 23:46:19,105 WARN >> org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while >> registering metric: numRecordsIn. >> java.lang.IllegalArgumentException: A metric named >> ip-99-9999-99-99.taskmanager.container_1596056409708_1570_01_000006.vdcs-kafka-flink-test.Map.0.numRecordsIn >> already exists >> at >> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >> 2020-08-26 23:46:19,094 WARN >> org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while >> registering metric: numRecordsOut. >> java.lang.IllegalArgumentException: A metric named >> ip-99-9999-99-999.taskmanager.container_1596056409708_1570_01_000005.vdcs-kafka-flink-test.Map.2.numRecordsOut >> already exists >> at >> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >> at >> org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) >> at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >> at >> org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) >> at >> org.apache.flink.runtime.metrics.MetricRegistryImpl.register(MetricRegistryImpl.java:343) >> at >> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.addMetric(AbstractMetricGroup.java:426) >> at >> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:359) >> at >> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:349) >> at >> org.apache.flink.runtime.metrics.groups.OperatorIOMetricGroup.<init>(OperatorIOMetricGroup.java:41) >> at >> org.apache.flink.runtime.metrics.groups.OperatorMetricGroup.<init>(OperatorMetricGroup.java:48) >> at >> org.apache.flink.runtime.metrics.groups.TaskMetricGroup.lambda$getOrAddOperator$0(TaskMetricGroup.java:154) >> at java.util.HashMap.computeIfAbsent(HashMap.java:1127) >> at >> org.apache.flink.runtime.metrics.groups.TaskMetricGroup.getOrAddOperator(TaskMetricGroup.java:154) >> at >> org.apache.flink.streaming.api.operators.AbstractStreamOperator.setup(AbstractStreamOperator.java:180) >> at >> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.setup(AbstractUdfStreamOperator.java:82) >> at >> org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:75) >> at >> org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:48) >> at >> org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:429) >> at >> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:353) >> at >> org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:144) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:433) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461) >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) >> at java.lang.Thread.run(Thread.java:748) >> Regards, >> Vijay >> >> >> On Wed, Aug 26, 2020 at 7:53 AM Chesnay Schepler <ches...@apache.org> >> wrote: >> >>> metrics.reporter.grph.class: >>> org.apache.flink.metrics.graphite.GraphiteReporter >>> >>> >>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphitegraphitereporter >>> >>> On 26/08/2020 16:40, Vijayendra Yadav wrote: >>> >>> Hi Dawid, >>> >>> I have 1.10.0 version of flink. What is alternative for this version ? >>> >>> Regards, >>> Vijay >>> >>> >>> On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz <dwysakow...@apache.org> >>> <dwysakow...@apache.org> wrote: >>> >>> >>> >>> Hi Vijay, >>> >>> I think the problem might be that you are using a wrong version of the >>> reporter. >>> >>> You say you used flink-metrics-graphite-1.10.0.jar from 1.10 as a >>> plugin, but it was migrated to plugins in 1.11 only[1]. >>> >>> I'd recommend trying it out with the same 1.11 version of Flink and >>> Graphite reporter. >>> >>> Best, >>> >>> Dawid >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-16965 >>> On 26/08/2020 08:04, Vijayendra Yadav wrote: >>> >>> Hi Nikola, >>> >>> To rule out any other cluster issues, I have tried it in my local now. >>> Steps as follows, but don't see any metrics yet. >>> >>> 1) Set up local Graphite >>> >>> docker run -d\ >>> --name graphite\ >>> --restart=always\ >>> -p 80:80\ >>> -p 2003-2004:2003-2004\ >>> -p 2023-2024:2023-2024\ >>> -p 8125:8125/udp\ >>> -p 8126:8126\ >>> graphiteapp/graphite-statsd >>> >>> Mapped Ports >>> Host Container Service >>> 80 80 nginx <https://www.nginx.com/resources/admin-guide/> >>> 2003 2003 carbon receiver - plaintext >>> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol> >>> 2004 2004 carbon receiver - pickle >>> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol> >>> 2023 2023 carbon aggregator - plaintext >>> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >>> 2024 2024 carbon aggregator - pickle >>> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >>> 8080 8080 Graphite internal gunicorn port (without Nginx proxying). >>> 8125 8125 statsd >>> <https://github.com/etsy/statsd/blob/master/docs/server.md> >>> 8126 8126 statsd admin >>> <https://github.com/etsy/statsd/blob/master/docs/admin_interface.md> >>> 2) WebUI: >>> >>> <image.png> >>> >>> >>> >>> 3) Run Flink example Job. >>> ./bin/flink run >>> ./examples/flink-examples-streaming_2.11-1.11-SNAPSHOT-SocketWindowWordCount.jar >>> --port 9999 >>> >>> with conf/flink-conf.yaml set as: >>> >>> metrics.reporter.grph.factory.class: >>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>> metrics.reporter.grph.host: localhost >>> metrics.reporter.grph.port: 2003 >>> metrics.reporter.grph.protocol: TCP >>> metrics.reporter.grph.interval: 1 SECONDS >>> >>> and graphite jar: >>> >>> plugins/flink-metrics-graphite/flink-metrics-graphite-1.10.0.jar >>> >>> >>> 4) Can't see any activity in webui graphite. >>> >>> >>> Could you review and let me know what is wrong here ? any other way you >>> suggest to be able to view the raw metrics data ? >>> Also, do you have sample metrics raw format, you can share from any >>> other project. >>> >>> Regards, >>> Vijay >>> >>> >>> >>> >>> On Sun, Aug 23, 2020 at 9:26 PM Nikola Hrusov <n.hru...@gmail.com> >>> wrote: >>> >>>> Hi Vijay, >>>> >>>> Your steps look correct to me. >>>> Perhaps you can double check that the graphite port you are sending is >>>> correct? THe default carbon port is 2003 and if you use the aggregator it >>>> is 2023. >>>> >>>> You should be able to see in both flink jobmanager and taskmanager that >>>> the metrics have been initialized with the config you have pasted. >>>> >>>> Regards >>>> , >>>> Nikola Hrusov >>>> >>>> >>>> On Mon, Aug 24, 2020 at 5:00 AM Vijayendra Yadav <contact....@gmail.com> >>>> wrote: >>>> >>>>> Hi Team, >>>>> >>>>> I am trying to export Flink stream default metrics using Graphite, >>>>> but I can't find it in the Graphite metrics console. Could you confirm >>>>> the >>>>> steps below are correct? >>>>> >>>>> *1) Updated flink-conf.yaml* >>>>> >>>>> metrics.reporter.grph.factory.class: >>>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>>>> metrics.reporter.grph.host: port >>>>> metrics.reporter.grph.port: 9109 >>>>> metrics.reporter.grph.protocol: TCP >>>>> metrics.reporter.grph.interval: 30 SECONDS >>>>> >>>>> 2) Added Graphite jar in plugin folder : >>>>> >>>>> ll */usr/lib/flink/plugins/metric/* >>>>> *flink-metrics-graphite-1.10.0.jar* >>>>> >>>>> 3) Looking metrics in graphite server: >>>>> >>>>> http://port:8080/metrics <http://10.108.58.63:8080/metrics> >>>>> >>>>> Note: No code change is done. >>>>> >>>>> Regards, >>>>> Vijay >>>>> >>>>> >>>>> >>>
