Hi Alexander,
From my side I still think it should be reasonable to have a jar that contains
the code that are running in the clients and also shipped to the cluster. Then
this jar could also be included in the shipping jar list.
For the second issue, similarly I think you may first build the project to get
the jar containing the code, then fill the path of the generated jar in to test
the submitting.
Best,
Yun
------------------Original Mail ------------------
Sender:Alexander Bagerman <bager...@gmail.com>
Send Date:Thu Oct 29 11:38:45 2020
Recipients:Yun Gao <yungao...@aliyun.com>
CC:Flink ML <user@flink.apache.org>
Subject:Re: How to deploy dynamically generated flink jobs?
I did try it but this option seems to be for a third party jar. In my case I
would need to specify/ship a jar that contains the code where job is being
constracted. I'm not clear of
1. how to point to the containg jar
2. how to test such a submission from my project running in Eclipse
Alex
On Wed, Oct 28, 2020 at 8:21 PM Yun Gao <yungao...@aliyun.com> wrote:
Hi Alexander,
The signature of the createRemoteEnvironment is
public static StreamExecutionEnvironment createRemoteEnvironment(
String host, int port, String... jarFiles);
Which could also ship the jars to execute to remote cluster. Could you have a
try to also pass the jar files to the remote environment ?
Best,
Yun
------------------------------------------------------------------
Sender:Alexander Bagerman<bager...@gmail.com>
Date:2020/10/29 10:43:16
Recipient:<user@flink.apache.org>
Theme:How to deploy dynamically generated flink jobs?
Hi,
I am trying to build a functionality to dynamically configure a flink job
(Java) in my code based on some additional metadata and submit it to a flink
running in a session cluster.
Flink version is 1.11.2
The problem I have is how to provide a packed job to the cluster. When I am
trying the following code
StreamExecutionEnvironment env =
StreamExecutionEnvironment.createRemoteEnvironment(hostName, hostPort);...
configuring job workflow here...env.execute(jobName);
I am getting ClassNotFoundException stating that code for my mapping functions
did not make it to the cluster. Which makes sense.
What would be the right way to deploy dynamically configured flink jobs which
are not packaged as a jar file but rather generated ad-hoc?
Thanks
