Hi Andreas, you should be able to discover running JobManagers using YarnClient (looking at the API, "ApplicationReport#getTrackingUrl()" [1] may be what you're looking for). I don't think it would be easy to have a generic support for this in Flink without making assumptions about the underlying infrastructure.
[1] https://hadoop.apache.org/docs/current/api/org/apache/hadoop/yarn/api/records/ApplicationReport.html#getTrackingUrl-- Best, D. On Tue, Aug 3, 2021 at 5:00 AM Yangze Guo <karma...@gmail.com> wrote: > From my understanding, what you want is actually a management system > for Flink jobs. I think it might be good to submit the job(with `flink > run`) and retrieve the WebUI in another process. > > Best, > Yangze Guo > > On Mon, Aug 2, 2021 at 10:39 PM Hailu, Andreas [Engineering] > <andreas.ha...@gs.com> wrote: > > > > Hi Yangze, sure! > > > > After a submitted Flink app is complete, our client app polls the > RESTful interface to pull job metrics -- operator start/end times, > duration, records + bytes read/written etc... All of these metrics are all > published to a database for analytical purposes, again both programmatic > and ad-hoc. > > > > There was no clear exposure of ClusterClient, so we had originally > worked around this by extending the CliFrontend class with a bit of a > façade class that grabbed the ClusterClient from the executeProgram() > method: > > > > @Override > > protected void executeProgram(PackagedProgram program, ClusterClient > client, int parallelism) throws ProgramMissingJobException, > ProgramInvocationException { > > logAndSysout("Starting execution of program"); > > System.setProperty(JOB_MANAGER_WEB_INTERFACE_PROPERTY, > client.getWebInterfaceURL()); // <- Used elsewhere in application > > ... > > } > > > > These metrics prove immensely valuable as they help us optimize > performance, diagnose issues, as well as predict resource requirements for > applications. > > > > // ah > > > > -----Original Message----- > > From: Yangze Guo <karma...@gmail.com> > > Sent: Sunday, August 1, 2021 10:38 PM > > To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com> > > Cc: user@flink.apache.org > > Subject: Re: Obtain JobManager Web Interface URL > > > > AFAIK, the ClusterClient should not be exposed through the public API. > > Would you like to explain your use case and why you need to get the web > UI programmatically? > > > > Best, > > Yangze Guo > > > > On Fri, Jul 30, 2021 at 9:54 PM Hailu, Andreas [Engineering] < > andreas.ha...@gs.com> wrote: > > > > > > Hello Yangze, thanks for responding. > > > > > > I'm attempting to perform this programmatically on YARN, so looking at > a log just won't do :) What's the appropriate way to get an instance of a > ClusterClient? Do you know of any examples I can look at? > > > > > > // ah > > > > > > -----Original Message----- > > > From: Yangze Guo <karma...@gmail.com> > > > Sent: Thursday, July 29, 2021 11:17 PM > > > To: Hailu, Andreas [Engineering] <andreas.ha...@ny.email.gs.com> > > > Cc: user@flink.apache.org > > > Subject: Re: Obtain JobManager Web Interface URL > > > > > > Hi, Hailu > > > > > > AFAIK, the ClusterClient#getWebInterfaceURL has been available since > 1.10. > > > > > > Regarding the JobManager web interface, it will be print in the logs > when staring a native Kubernetes or Yarn cluster. In standalone mode, it is > configured by yourself[1]. > > > > > > [1] > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__ci.apache.org_pro > > > jects_flink_flink-2Ddocs-2Dmaster_docs_deployment_resource-2Dproviders > > > _standalone_overview_-23starting-2Dand-2Dstopping-2Da-2Dcluster&d=DwIF > > > aQ&c=7563p3e2zaQw0AB1wrFVgyagb2IE5rTZOYPxLxfZlX4&r=hRr4SA7BtUvKoMBP6VD > > > hfisy2OJ1ZAzai-pcCC6TFXM&m=FVv2XIIuWzaAGdj6tz9whXTJ5GQ_xgAqIgesdgtEjG4 > > > &s=Cu-w4-hIu8MGtvnq2Ob8StpWCZhbFmwN4knnt35NqOM&e= > > > > > > Best, > > > Yangze Guo > > > > > > On Fri, Jul 30, 2021 at 1:41 AM Hailu, Andreas [Engineering] < > andreas.ha...@gs.com> wrote: > > > > > > > > Hi team, > > > > > > > > > > > > > > > > Is there a method available to obtain the JobManager’s REST url? We > originally overloaded CliFrontend#executeProgram and nabbed it from the > ClusterClient#getWebInterfaceUrl method, but it seems this method’s > signature has been changed and no longer available as of 1.10.0. > > > > > > > > > > > > > > > > Best, > > > > > > > > Andreas > > > > > > > > > > > > > > > > > > > > ________________________________ > > > > > > > > Your Personal Data: We may collect and process information about you > > > > that may be subject to data protection laws. For more information > > > > about how we use and disclose your personal data, how we protect > > > > your information, our legal basis to use your information, your > > > > rights and who you can contact, please refer to: > > > > http://www.gs.com/privacy-notices > > > > > > ________________________________ > > > > > > Your Personal Data: We may collect and process information about you > > > that may be subject to data protection laws. For more information > > > about how we use and disclose your personal data, how we protect your > > > information, our legal basis to use your information, your rights and > > > who you can contact, please refer to: > > > www.gs.com/privacy-notices<http://www.gs.com/privacy-notices> > > > > ________________________________ > > > > Your Personal Data: We may collect and process information about you > that may be subject to data protection laws. For more information about how > we use and disclose your personal data, how we protect your information, > our legal basis to use your information, your rights and who you can > contact, please refer to: www.gs.com/privacy-notices< > http://www.gs.com/privacy-notices> >
