Hi Sachin, Happy new year ⊠đ
On Yarn we setup prometheus push gateway on one machine and the have al task managers export their monitoring to this push gateway. Then Prometheus would collect all metrics from push gateway. On yarn, the main problem is missing isolation of workloads and thus you easily run into port conflicts ⊠Hope this helps Thias From: Sachin Mittal <sjmit...@gmail.com> Sent: Monday, January 6, 2025 10:42 AM To: user <user@flink.apache.org> Subject: [External] Is there a way to get consolidated metrics at task level â EXTERNAL MESSAGE â CAUTION: Think Before You Click â Hello, I am running a Flink streaming pipeline with autscaler on AWS EMR (as a YARN application). My tasks are scaling independently based on load. My question is how can I get the metric, say numRecordsInPerSecond, at the task level. Basically it should sum all the values for this metric at sub task level and provide my once consolidated value. This way I can provide uniform reporting irrespective of the number of sub tasks that get created at different points in time. Thanks Sachin Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und beinhaltet unter UmstĂ€nden vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewĂ€hrleistet werden kann, ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller AnhĂ€nge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten. This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
