jiguodai created FLINK-21309:
--------------------------------
Summary: Metrics of JobManager and TaskManager overwrite each
other in pushgateway
Key: FLINK-21309
URL: https://issues.apache.org/jira/browse/FLINK-21309
Project: Flink
Issue Type: Bug
Components: Runtime / Metrics
Affects Versions: 1.11.0, 1.10.0, 1.9.0
Environment: 1. Components :
Flink 1.9.0/1.10.0/1.11.0 + Prometheus + Pushgateway + Yarn
2. Metrics Configuration in flink-conf.yaml :
{code:java}
metrics.reporter.promgateway.class:
org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
metrics.reporter.promgateway.jobName: myjob
metrics.reporter.promgateway.randomJobNameSuffix: false{code}
Reporter: jiguodai
Attachments: image-2021-02-05-21-07-42-292.png
When a flink job run on yarn, metrics of jobmanager and taskmanagers will
overwrite each other. The phenomenon is that on one second you can find only
jobmanager metrics on pushgateway web ui, while on the next second you can
find only taskmanager metrics on pushgateway web ui, these two kinds of metrics
appear alternately. One metric of taskmanager on grafana will be like below
intermittently (this taskmanager metric disappear on grafana when jobmanager
metrics overwrite taskmanager metrics):
!image-2021-02-05-21-07-42-292.png!
The real reason is that Flink PrometheusPushGatewayReporter use PUT style
instead of POST style to push metrics to pushgateway, what's more, taskmanagers
and jobmanager use the same jobName (the only grouping key) which we configured
in flink-conf.yaml.
Althought REST URLs are same as below,
{code:java}
/metrics/job/<JOB_NAME>{/<LABEL_NAME>/<LABEL_VALUE>}
{code}
PUT and POST caused different results, as we can see below :
* PUT is used to push a group of metrics. All metrics with the grouping key
specified in the URL are replaced by the metrics pushed with PUT.
* POST works exactly like the PUT method but only metrics with the same name
as the newly pushed metrics are replaced.
For these reasons, it's better to use POST style to push metrics to
pushgateway to prevent jobmanager metrics and taskmanager metrics from
overwriting each other, so that we can get continuous graph on grafana.
Everytime some body ask why metrics on grafana is discontinuous on Flink
mailing list, i will tell him that you should change the style of pushing
metrics to pushgateway from PUT to POST and then repackage the
flink-metrics-prometheus module. So, why don't we solve the problem permanently
now ? I hope to have the chance to solve the problem, sincerely.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
