Hi Till, *Info below, also I have a question at the end. * pretty much what was told earlier, for 1.10.0 use: metrics.reporter.grph.class: org.apache.flink.metrics.graphite.GraphiteReporter And then Looks like my Graphite installation has a problem or I am missing something. Just to make sure Flink exporter has no problem I set up graphite exporter config to local host and port 8888 and then i started listening to port using netcat and I was able to see metrics:
vy0769@ubuntu:~/IdeaProjects/vdcs-kafka-flink-ingestion$* nc -lv -p 8888* Listening on [0.0.0.0] (family 0, port 8888) *localhost.jobmanager.Socket-Window-WordCount.downtime 0 1599063945localhost.jobmanager.Socket-Window-WordCount.fullRestarts 0 1599063945localhost.jobmanager.Socket-Window-WordCount.lastCheckpointAlignmentBuffered -1 1599063945* *I have a Question above: There two values say: 0 1599063945. Is value 1599063945 the Timestamp ?* Regards, Vijay On Wed, Sep 2, 2020 at 2:06 AM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Vijayendra, > > what was the problem in the end? Sharing this knowledge might help other > community member who might run into the same issue. > > Cheers, > Till > > On Wed, Sep 2, 2020 at 3:34 AM Vijayendra Yadav <contact....@gmail.com> > wrote: > >> Thanks all, I could see the metrics. >> >> On Thu, Aug 27, 2020 at 7:51 AM Robert Metzger <rmetz...@apache.org> >> wrote: >> >>> I don't think these error messages give us a hint why you can't see the >>> metrics (because they are about registering metrics, not reporting them) >>> >>> Are you sure you are using the right configuration parameters for Flink >>> 1.10? That all required JARs are in the lib/ folder (on all machines) and >>> that your graphite setup is working (have you confirmed that you can show >>> any metrics in the Graphite UI (maybe from a Graphite demo thingy))? >>> >>> >>> On Thu, Aug 27, 2020 at 2:05 AM Vijayendra Yadav <contact....@gmail.com> >>> wrote: >>> >>>> Hi Chesnay and Dawid, >>>> >>>> I see multiple entries as following in Log: >>>> >>>> 2020-08-26 23:46:19,105 WARN >>>> org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while >>>> registering metric: numRecordsIn. >>>> java.lang.IllegalArgumentException: A metric named >>>> ip-99-9999-99-99.taskmanager.container_1596056409708_1570_01_000006.vdcs-kafka-flink-test.Map.0.numRecordsIn >>>> already exists >>>> at >>>> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >>>> 2020-08-26 23:46:19,094 WARN >>>> org.apache.flink.runtime.metrics.MetricRegistryImpl - Error while >>>> registering metric: numRecordsOut. >>>> java.lang.IllegalArgumentException: A metric named >>>> ip-99-9999-99-999.taskmanager.container_1596056409708_1570_01_000005.vdcs-kafka-flink-test.Map.2.numRecordsOut >>>> already exists >>>> at >>>> com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >>>> at >>>> org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) >>>> at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91) >>>> at >>>> org.apache.flink.dropwizard.ScheduledDropwizardReporter.notifyOfAddedMetric(ScheduledDropwizardReporter.java:131) >>>> at >>>> org.apache.flink.runtime.metrics.MetricRegistryImpl.register(MetricRegistryImpl.java:343) >>>> at >>>> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.addMetric(AbstractMetricGroup.java:426) >>>> at >>>> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:359) >>>> at >>>> org.apache.flink.runtime.metrics.groups.AbstractMetricGroup.counter(AbstractMetricGroup.java:349) >>>> at >>>> org.apache.flink.runtime.metrics.groups.OperatorIOMetricGroup.<init>(OperatorIOMetricGroup.java:41) >>>> at >>>> org.apache.flink.runtime.metrics.groups.OperatorMetricGroup.<init>(OperatorMetricGroup.java:48) >>>> at >>>> org.apache.flink.runtime.metrics.groups.TaskMetricGroup.lambda$getOrAddOperator$0(TaskMetricGroup.java:154) >>>> at java.util.HashMap.computeIfAbsent(HashMap.java:1127) >>>> at >>>> org.apache.flink.runtime.metrics.groups.TaskMetricGroup.getOrAddOperator(TaskMetricGroup.java:154) >>>> at >>>> org.apache.flink.streaming.api.operators.AbstractStreamOperator.setup(AbstractStreamOperator.java:180) >>>> at >>>> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.setup(AbstractUdfStreamOperator.java:82) >>>> at >>>> org.apache.flink.streaming.api.operators.SimpleOperatorFactory.createStreamOperator(SimpleOperatorFactory.java:75) >>>> at >>>> org.apache.flink.streaming.api.operators.StreamOperatorFactoryUtil.createOperator(StreamOperatorFactoryUtil.java:48) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createChainedOperator(OperatorChain.java:429) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:353) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:144) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:433) >>>> at >>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461) >>>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) >>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) >>>> at java.lang.Thread.run(Thread.java:748) >>>> Regards, >>>> Vijay >>>> >>>> >>>> On Wed, Aug 26, 2020 at 7:53 AM Chesnay Schepler <ches...@apache.org> >>>> wrote: >>>> >>>>> metrics.reporter.grph.class: >>>>> org.apache.flink.metrics.graphite.GraphiteReporter >>>>> >>>>> >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.10/monitoring/metrics.html#graphite-orgapacheflinkmetricsgraphitegraphitereporter >>>>> >>>>> On 26/08/2020 16:40, Vijayendra Yadav wrote: >>>>> >>>>> Hi Dawid, >>>>> >>>>> I have 1.10.0 version of flink. What is alternative for this version ? >>>>> >>>>> Regards, >>>>> Vijay >>>>> >>>>> >>>>> On Aug 25, 2020, at 11:44 PM, Dawid Wysakowicz >>>>> <dwysakow...@apache.org> <dwysakow...@apache.org> wrote: >>>>> >>>>> >>>>> >>>>> Hi Vijay, >>>>> >>>>> I think the problem might be that you are using a wrong version of the >>>>> reporter. >>>>> >>>>> You say you used flink-metrics-graphite-1.10.0.jar from 1.10 as a >>>>> plugin, but it was migrated to plugins in 1.11 only[1]. >>>>> >>>>> I'd recommend trying it out with the same 1.11 version of Flink and >>>>> Graphite reporter. >>>>> >>>>> Best, >>>>> >>>>> Dawid >>>>> >>>>> [1] https://issues.apache.org/jira/browse/FLINK-16965 >>>>> On 26/08/2020 08:04, Vijayendra Yadav wrote: >>>>> >>>>> Hi Nikola, >>>>> >>>>> To rule out any other cluster issues, I have tried it in my local now. >>>>> Steps as follows, but don't see any metrics yet. >>>>> >>>>> 1) Set up local Graphite >>>>> >>>>> docker run -d\ >>>>> --name graphite\ >>>>> --restart=always\ >>>>> -p 80:80\ >>>>> -p 2003-2004:2003-2004\ >>>>> -p 2023-2024:2023-2024\ >>>>> -p 8125:8125/udp\ >>>>> -p 8126:8126\ >>>>> graphiteapp/graphite-statsd >>>>> >>>>> Mapped Ports >>>>> Host Container Service >>>>> 80 80 nginx <https://www.nginx.com/resources/admin-guide/> >>>>> 2003 2003 carbon receiver - plaintext >>>>> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-plaintext-protocol> >>>>> 2004 2004 carbon receiver - pickle >>>>> <http://graphite.readthedocs.io/en/latest/feeding-carbon.html#the-pickle-protocol> >>>>> 2023 2023 carbon aggregator - plaintext >>>>> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >>>>> 2024 2024 carbon aggregator - pickle >>>>> <http://graphite.readthedocs.io/en/latest/carbon-daemons.html#carbon-aggregator-py> >>>>> 8080 8080 Graphite internal gunicorn port (without Nginx proxying). >>>>> 8125 8125 statsd >>>>> <https://github.com/etsy/statsd/blob/master/docs/server.md> >>>>> 8126 8126 statsd admin >>>>> <https://github.com/etsy/statsd/blob/master/docs/admin_interface.md> >>>>> 2) WebUI: >>>>> >>>>> <image.png> >>>>> >>>>> >>>>> >>>>> 3) Run Flink example Job. >>>>> ./bin/flink run >>>>> ./examples/flink-examples-streaming_2.11-1.11-SNAPSHOT-SocketWindowWordCount.jar >>>>> --port 9999 >>>>> >>>>> with conf/flink-conf.yaml set as: >>>>> >>>>> metrics.reporter.grph.factory.class: >>>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>>>> metrics.reporter.grph.host: localhost >>>>> metrics.reporter.grph.port: 2003 >>>>> metrics.reporter.grph.protocol: TCP >>>>> metrics.reporter.grph.interval: 1 SECONDS >>>>> >>>>> and graphite jar: >>>>> >>>>> plugins/flink-metrics-graphite/flink-metrics-graphite-1.10.0.jar >>>>> >>>>> >>>>> 4) Can't see any activity in webui graphite. >>>>> >>>>> >>>>> Could you review and let me know what is wrong here ? any other way >>>>> you suggest to be able to view the raw metrics data ? >>>>> Also, do you have sample metrics raw format, you can share from any >>>>> other project. >>>>> >>>>> Regards, >>>>> Vijay >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, Aug 23, 2020 at 9:26 PM Nikola Hrusov <n.hru...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Vijay, >>>>>> >>>>>> Your steps look correct to me. >>>>>> Perhaps you can double check that the graphite port you are sending >>>>>> is correct? THe default carbon port is 2003 and if you use the aggregator >>>>>> it is 2023. >>>>>> >>>>>> You should be able to see in both flink jobmanager and taskmanager >>>>>> that the metrics have been initialized with the config you have pasted. >>>>>> >>>>>> Regards >>>>>> , >>>>>> Nikola Hrusov >>>>>> >>>>>> >>>>>> On Mon, Aug 24, 2020 at 5:00 AM Vijayendra Yadav < >>>>>> contact....@gmail.com> wrote: >>>>>> >>>>>>> Hi Team, >>>>>>> >>>>>>> I am trying to export Flink stream default metrics using Graphite, >>>>>>> but I can't find it in the Graphite metrics console. Could you confirm >>>>>>> the >>>>>>> steps below are correct? >>>>>>> >>>>>>> *1) Updated flink-conf.yaml* >>>>>>> >>>>>>> metrics.reporter.grph.factory.class: >>>>>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory >>>>>>> metrics.reporter.grph.host: port >>>>>>> metrics.reporter.grph.port: 9109 >>>>>>> metrics.reporter.grph.protocol: TCP >>>>>>> metrics.reporter.grph.interval: 30 SECONDS >>>>>>> >>>>>>> 2) Added Graphite jar in plugin folder : >>>>>>> >>>>>>> ll */usr/lib/flink/plugins/metric/* >>>>>>> *flink-metrics-graphite-1.10.0.jar* >>>>>>> >>>>>>> 3) Looking metrics in graphite server: >>>>>>> >>>>>>> http://port:8080/metrics <http://10.108.58.63:8080/metrics> >>>>>>> >>>>>>> Note: No code change is done. >>>>>>> >>>>>>> Regards, >>>>>>> Vijay >>>>>>> >>>>>>> >>>>>>> >>>>>
