It is currently not possible to select metrics.
What you can do however is create a custom reporter that wraps the
StatsD reporter which does this filtering.
On 16/02/2022 17:41, Diwakar Jha wrote:
Hello,
Could someone please help! I'm trying to publish only these three
metrics per tasknode
Status.JVM.Memory.Heap.Used
Status.JVM.Memory.Heap.Committed
Status.JVM.Memory.NonHeap.Max
But, with my current setting I see all Flink metrics getting
published. Please let me know if I need to provide any other information.
Thank you!
---------- Forwarded message ---------
From: *Diwakar Jha* <diwakar.n...@gmail.com>
Date: Tue, Feb 15, 2022 at 1:31 PM
Subject: How to get memory specific metrics for tasknodes
To: user <user@flink.apache.org>
Hello,
I'm running Flink 1.11 on AWS EMR using the Yarn application. I'm
trying to access memory metrics(Heap.Max, Heap.Used) per tasknode in
CloudWatch. I have 50 tasknodes and it creates Millions of
metrics(including per operator) though I need only a few metrics per
tasknode (Heap.Max, Heap.Used). It is way too much than my current
cloudwatch limit and I also don't need so many metrics.
Could someone please help me how to get only the tasknode memory
specific metrics ?
I'm referring to this doc :
https://nightlies.apache.org/flink/flink-docs-release-1.7/monitoring/metrics.html#memory
I used the following approach to enable Flink metrics.
1. Enable Flink Metrics
copy |/opt/flink-metrics-statsd-x.x.jar| into the |/lib| folder of
your Flink distribution
2. Add StatsD metric reporter in Flink-conf to send to CloudWatch
Agent's StatsD interface
metrics.reporters: stsd
metrics.reporter.stsd.factory.class:
org.apache.flink.metrics.statsd.StatsDReporterFactory
metrics.reporter.stsd.host: localhost
metrics.reporter.stsd.port: 8125
3. Setup tasknode scope
metrics.scope.tm <http://metrics.scope.tm>: taskmanager
4. setup Cloudwatch agent to publish the metrics
"metrics":{
"namespace": "CustomeNamespace/FlinkMemoryMetrics",
"metrics_collected":{
"statsd":{
"service_address":":8125",
"metrics_collection_interval":60,
"metrics_aggregation_interval":300
}
}
},
Thanks!
