Hi Jessy,
to be precise, the JobGraph is not used at runtime. It is translated
into an ExecutionGraph.
But nevertheless such patterns are possible but require a bit of manual
implementation.
Option 1) You stop the job with a savepoint and restart the application
with slightly different parameters. If the pipeline has not changed
much, the old state can be remapped to the slightly modified job graph.
This is the easiest solution but with the downside of maybe a couple of
seconds downtime.
Option 2) You introduce a dedicated control stream (i.e. by using the
connect() DataStream API [1]). Either you implement a custom sink in the
main stream of the CoProcessFunction. Or you enrich every record in the
main stream with sink parameters that are read by you custom sink
implementation.
I hope this helps.
Regards,
Timo
[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/operators/overview/#connect
On 16.03.21 12:37, Jessy Ping wrote:
Hi Team,
Is it possible to edit the job graph at runtime ? . Suppose, I want to
add a new sink to the flink application at runtime that depends upon
the specific parameters in the incoming events.Can i edit the jobgraph
of a running flink application ?
Thanks
Jessy
