Or is there a way to specify in the command line to have the jm and tm run in different containers on YARN?
On Thu, Aug 30, 2018 at 11:51 PM Deirdre Kong <deirdre.y.k...@gmail.com> wrote: > @Chesnay, can you elaborate on how to map specific ports to a specific > JM/TM process? > > @Vino, I can only update Prometheus configuration once. Say I set my port > to be 9249-9250 in flink-conf.yml, and configure Prometheus to listen on > <JM-IP>:9249 for JM metrics and <TM-IP>:9250 for TM metrics. If JM and TM > are deployed in the same container, then I have no issue. But if YARN > deployed them in different containers then the TM metrics will expose on > port 9249 instead. > > Thanks, > Deirdre > > On Thu, Aug 30, 2018 at 11:47 PM Chesnay Schepler <ches...@apache.org> > wrote: > >> I don't know how/whether you can provide different flink-conf.yaml files >> when using YARN. >> >> But this is the only way to map specific ports to a specific JM/TM >> process. >> >> On 31.08.2018 08:15, vino yang wrote: >> >> Hi Deirdre, >> >> Sorry, I thought you were using a Standalone cluster environment. >> If you are based on YARN, then it seems that the way I said does not >> work. Maybe you specify a larger port range and will not cause conflicts. >> I am curious, why do you want to fix the run port of the reporter? >> >> Maybe you can expect Chesnay to give you advice. >> >> Thanks, vino. >> >> Deirdre Kong <deirdre.y.k...@gmail.com> 于2018年8月31日周五 下午1:26写道： >> >>> Hi Vino/Chesnay, >>> >>> Thank you for the info. >>> >>> I am actually using Yarn for deployment. Flink is installed in AWS EMR, >>> so sometimes jm and tm processes are deployed in the same container, >>> sometimes they are deployed in different containers. I would need to >>> configure Prometheus to listen on a specify port for JM and TM reporter. >>> So even if JM is bounced, I can get the JM metrics on the same port each >>> time. >>> >>> Can you elaborate what do you mean by specify a separate Flink >>> installation package for it? >>> >>> Chesnay, do you have any insights on this? >>> >>> Thanks, >>> Deirdre >>> >>> On Thu, Aug 30, 2018 at 7:18 PM vino yang <yanghua1...@gmail.com> wrote: >>> >>>> Hi Deirdre, >>>> >>>> If you run multiple Flink component (jm/tm) processes on one physical >>>> node, it is recommended to specify the port range to avoid conflicts[1], I >>>> guess this is based on the same Flink binary installation package. >>>> If you want to always have the TM reporter running on the same port, >>>> you can specify a separate Flink installation package for it, explicitly >>>> specifying this port in the configuration file for this installation >>>> package. >>>> However, you still need to pay attention to port conflicts with other >>>> components. >>>> >>>> The issue you provided is handled by Chesnay, so maybe Chesnay opinion >>>> is more appropriate. Ping Chesnay for you. >>>> >>>> [1]: >>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter >>>> >>>> Thanks, vino. >>>> >>>> Deirdre Kong <deirdre.y.k...@gmail.com> 于2018年8月31日周五 上午5:25写道： >>>> >>>>> Hi, >>>>> >>>>> I am running Flink in Amazon EMR. In flink-conf.yaml, I have >>>>> `metrics.reporter.prom.port: >>>>> 9249-9250` >>>>> Depending whether the job manager and task manager are running in the >>>>> same node, the task manager metrics are reported on port 9250 (if running >>>>> on same node as job manager), or on port 9249 (if running on a different >>>>> node). >>>>> >>>>> Is there a way to configure so that the task manager metrics are >>>>> always reported on port 9250? >>>>> I saw a post <https://issues.apache.org/jira/browse/FLINK-8284> that >>>>> we can "provide each *Manager with a separate configuration." How to do >>>>> that? >>>>> >>>>> Thanks >>>>> >>>> >>
