I'm surprised that this is not possible currently. Seems like a glaring missing feature to me.
I'd assume the best way would be to extend the REST API to /jar/:jarId/run with an option to overwrite configuration values. I'm not sure how to map json well to the yaml structure of the config, but I guess we mostly have simple key/value pairs anyways. On Tue, Dec 8, 2020 at 1:31 PM Till Rohrmann <trohrm...@apache.org> wrote: > Hi Fabian, > > thanks for starting this discussion. In general I would be a bit hesitant > to build upon Flink's web UI submission because it suffers from a couple of > drawbacks. > > 1) The web UI submission only supports single job applications. > 2) The JobGraph is generated from within the web UI Netty thread. Hence, if > the user code blocks, then this can make the web UI unresponsive. > 3) Uploaded jars are not persisted. Hence, if a JobManager failover occurs > between uploading and running the job, then you might have lost the > uploaded jars. > > The reason for some of these problems is that the feature was actually > implemented for some conference and almost remained untouched ever since. > Building more functionality on top of it will mean that it will be harder > to remove in the future. > > Cheers, > Till > > On Tue, Dec 8, 2020 at 12:00 PM Fabian Paul <fabianp...@data-artisans.com> > wrote: > > > Hi all, > > > > Currently, the most convenient way of programmatically submitting a job > to > > a running session cluster is using Flink’s RestClusterClient. > > Unfortunately, it is only supported, as of now, to submit a job graph.[1] > > To construct a job graph from a jar file, additional Flink dependencies > are > > required, which is not ideal. > > > > It is also possible to directly use the Flink rest API and first upload > > the jar file via /jars/upload[2] and then run it via /jar/:jarId/run[3]. > It > > has the downside that it is impossible to set a Flink execution > > configuration, and it will always take the underlying session cluster > > configuration. > > > > I know changing the ClusterClient has already been discussed. It would > > involve a lot of effort, so what do you think of making the jar execution > > more prominent via the rest endpoint by adding the option to pass an > > execution configuration? > > > > Best, > > Fabian > > > > [1] > > > https://github.com/apache/flink/blob/65cd385d7de504a946b17193aceea73b3c8e78a8/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java#L95 > > [2] > > > https://github.com/apache/flink/blob/c2972b6e336cc3b3a6cbd22c69a6710dab5246e6/flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterConfigurationParserFactory.java#L56 > > < > > > https://github.com/apache/flink/blob/c2972b6e336cc3b3a6cbd22c69a6710dab5246e6/flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterConfigurationParserFactory.java#L56 > > > > > [3] > > > https://github.com/apache/flink/blob/c2972b6e336cc3b3a6cbd22c69a6710dab5246e6/flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterConfigurationParserFactory.java#L56 > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
