Hi Deirdre, Usually, we don't recommend JM and TM in a container. @Chesnay, right? I want to confirm, is your container here meaning node?
Thanks, vino. Deirdre Kong <deirdre.y.k...@gmail.com> 于2018年8月31日周五 下午3:03写道： > Or is there a way to specify in the command line to have the jm and tm run > in different containers on YARN? > > On Thu, Aug 30, 2018 at 11:51 PM Deirdre Kong <deirdre.y.k...@gmail.com> > wrote: > >> @Chesnay, can you elaborate on how to map specific ports to a specific >> JM/TM process? >> >> @Vino, I can only update Prometheus configuration once. Say I set my >> port to be 9249-9250 in flink-conf.yml, and configure Prometheus to listen >> on <JM-IP>:9249 for JM metrics and <TM-IP>:9250 for TM metrics. If JM and >> TM are deployed in the same container, then I have no issue. But if YARN >> deployed them in different containers then the TM metrics will expose on >> port 9249 instead. >> >> Thanks, >> Deirdre >> >> On Thu, Aug 30, 2018 at 11:47 PM Chesnay Schepler <ches...@apache.org> >> wrote: >> >>> I don't know how/whether you can provide different flink-conf.yaml files >>> when using YARN. >>> >>> But this is the only way to map specific ports to a specific JM/TM >>> process. >>> >>> On 31.08.2018 08:15, vino yang wrote: >>> >>> Hi Deirdre, >>> >>> Sorry, I thought you were using a Standalone cluster environment. >>> If you are based on YARN, then it seems that the way I said does not >>> work. Maybe you specify a larger port range and will not cause conflicts. >>> I am curious, why do you want to fix the run port of the reporter? >>> >>> Maybe you can expect Chesnay to give you advice. >>> >>> Thanks, vino. >>> >>> Deirdre Kong <deirdre.y.k...@gmail.com> 于2018年8月31日周五 下午1:26写道： >>> >>>> Hi Vino/Chesnay, >>>> >>>> Thank you for the info. >>>> >>>> I am actually using Yarn for deployment. Flink is installed in AWS >>>> EMR, so sometimes jm and tm processes are deployed in the same container, >>>> sometimes they are deployed in different containers. I would need to >>>> configure Prometheus to listen on a specify port for JM and TM reporter. >>>> So even if JM is bounced, I can get the JM metrics on the same port each >>>> time. >>>> >>>> Can you elaborate what do you mean by specify a separate Flink >>>> installation package for it? >>>> >>>> Chesnay, do you have any insights on this? >>>> >>>> Thanks, >>>> Deirdre >>>> >>>> On Thu, Aug 30, 2018 at 7:18 PM vino yang <yanghua1...@gmail.com> >>>> wrote: >>>> >>>>> Hi Deirdre, >>>>> >>>>> If you run multiple Flink component (jm/tm) processes on one physical >>>>> node, it is recommended to specify the port range to avoid conflicts[1], I >>>>> guess this is based on the same Flink binary installation package. >>>>> If you want to always have the TM reporter running on the same port, >>>>> you can specify a separate Flink installation package for it, explicitly >>>>> specifying this port in the configuration file for this installation >>>>> package. >>>>> However, you still need to pay attention to port conflicts with other >>>>> components. >>>>> >>>>> The issue you provided is handled by Chesnay, so maybe Chesnay opinion >>>>> is more appropriate. Ping Chesnay for you. >>>>> >>>>> [1]: >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/metrics.html#prometheus-orgapacheflinkmetricsprometheusprometheusreporter >>>>> >>>>> Thanks, vino. >>>>> >>>>> Deirdre Kong <deirdre.y.k...@gmail.com> 于2018年8月31日周五 上午5:25写道： >>>>> >>>>>> Hi, >>>>>> >>>>>> I am running Flink in Amazon EMR. In flink-conf.yaml, I have >>>>>> `metrics.reporter.prom.port: >>>>>> 9249-9250` >>>>>> Depending whether the job manager and task manager are running in the >>>>>> same node, the task manager metrics are reported on port 9250 (if running >>>>>> on same node as job manager), or on port 9249 (if running on a different >>>>>> node). >>>>>> >>>>>> Is there a way to configure so that the task manager metrics are >>>>>> always reported on port 9250? >>>>>> I saw a post <https://issues.apache.org/jira/browse/FLINK-8284> that >>>>>> we can "provide each *Manager with a separate configuration." How to do >>>>>> that? >>>>>> >>>>>> Thanks >>>>>> >>>>> >>>
