Hi Sachin again, In case of Prometheus it’s done by means of the query against it.
* See aggregation ops [1] * Each metric has properties (like dimensions) and you can aggregate over them * See examples here [2] [1] https://prometheus.io/docs/prometheus/latest/querying/operators/#aggregation-operators [2] https://prometheus.io/docs/prometheus/latest/querying/examples/#using-functions-operators-etc Thias From: Sachin Mittal <sjmit...@gmail.com> Sent: Monday, January 6, 2025 3:23 PM To: Schwalbe Matthias <matthias.schwa...@viseca.ch> Cc: user <user@flink.apache.org> Subject: [External] Re: Is there a way to get consolidated metrics at task level ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ So how can we aggregate same metric across all the sub tasks ? Say: Task_0 is outputting a value for numRecordsInPerSecond Task_1 is outputting a value for numRecordsInPerSecond I want to consolidate numRecordsInPerSecond for the entire task which would be the sum of these values for all the number of subtasks that are running which can vary with time. Thanks Sachin On Mon, Jan 6, 2025 at 4:09 PM Schwalbe Matthias <matthias.schwa...@viseca.ch<mailto:matthias.schwa...@viseca.ch>> wrote: Hi Sachin, Happy new year … 😊 On Yarn we setup prometheus push gateway on one machine and the have al task managers export their monitoring to this push gateway. Then Prometheus would collect all metrics from push gateway. On yarn, the main problem is missing isolation of workloads and thus you easily run into port conflicts … Hope this helps Thias From: Sachin Mittal <sjmit...@gmail.com<mailto:sjmit...@gmail.com>> Sent: Monday, January 6, 2025 10:42 AM To: user <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: [External] Is there a way to get consolidated metrics at task level ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ Hello, I am running a Flink streaming pipeline with autscaler on AWS EMR (as a YARN application). My tasks are scaling independently based on load. My question is how can I get the metric, say numRecordsInPerSecond, at the task level. Basically it should sum all the values for this metric at sub task level and provide my once consolidated value. This way I can provide uniform reporting irrespective of the number of sub tasks that get created at different points in time. Thanks Sachin Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited. Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
