As I said, this is not possible. In the custom reporter you can filter out all the metrics you do not want.

On 18/02/2022 19:44, Diwakar Jha wrote:
Thank you. I understand that filtering metrics is not possible but i can configure it for only the memory metrics listed here? : https://nightlies.apache.org/flink/flink-docs-master/docs/ops/metrics/#memory Currently, it's publishing all the container/operator metrics which is what I want to avoid. Thanks!


On Wed, Feb 16, 2022 at 11:35 PM Chesnay Schepler <ches...@apache.org> wrote:

    It is currently not possible to select metrics.

    What you can do however is create a custom reporter that wraps the
    StatsD reporter which does this filtering.

    On 16/02/2022 17:41, Diwakar Jha wrote:

    Hello,

    Could someone please help! I'm trying  to publish only these
    three metrics per tasknode
    Status.JVM.Memory.Heap.Used
    Status.JVM.Memory.Heap.Committed
    Status.JVM.Memory.NonHeap.Max

    But, with my current setting I see all Flink metrics getting
    published. Please let me know if I need to provide any other
    information.

    Thank you!


    ---------- Forwarded message ---------
    From: *Diwakar Jha* <diwakar.n...@gmail.com>
    Date: Tue, Feb 15, 2022 at 1:31 PM
    Subject: How to get memory specific metrics for tasknodes
    To: user <user@flink.apache.org>


    Hello,

    I'm running Flink 1.11 on AWS EMR using the Yarn application. I'm
    trying to access memory metrics(Heap.Max, Heap.Used) per
    tasknode in CloudWatch. I have 50 tasknodes and it creates
    Millions of metrics(including per operator) though I need only a
    few metrics per tasknode (Heap.Max, Heap.Used). It is way too
    much than my current cloudwatch limit and I also don't need so
    many metrics.
    Could someone please help me how to get only the tasknode memory
    specific metrics ?
    I'm referring to this doc :
    
https://nightlies.apache.org/flink/flink-docs-release-1.7/monitoring/metrics.html#memory

    I used the following approach to enable Flink metrics.
    1. Enable Flink Metrics
    copy |/opt/flink-metrics-statsd-x.x.jar| into the |/lib| folder
    of your Flink distribution
    2.  Add StatsD metric reporter in Flink-conf to send to
    CloudWatch Agent's StatsD interface
                metrics.reporters: stsd
                metrics.reporter.stsd.factory.class:
    org.apache.flink.metrics.statsd.StatsDReporterFactory
                metrics.reporter.stsd.host: localhost
                metrics.reporter.stsd.port: 8125
    3. Setup tasknode scope
    metrics.scope.tm <http://metrics.scope.tm>: taskmanager
    4. setup Cloudwatch agent to publish the metrics
    "metrics":{
                  "namespace": "CustomeNamespace/FlinkMemoryMetrics",
                  "metrics_collected":{
                     "statsd":{
                        "service_address":":8125",
    "metrics_collection_interval":60,
    "metrics_aggregation_interval":300
                     }
                  }
              },

    Thanks!



Reply via email to